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PLURIBUS DOCUMENT 2: SYSTEM HANDBOOK 
PREFACE 


"Pluribus Document 2: System Handbook" is one of a series 

which taken together provides complete documentation of the 
Pluribus line of computer systems. In the present document, 
Part 1, entitled "System Description," contains an extensive 
discussion of the Pluribus line and the ways in which it can 

be used. This system description is the primary text for anyone 
seeking familiarity with the Pluribus, although, of course, 
there are many details which can only be found elsewhere in the 
set. Part 2 is a glossary of specialized Pluribus terms used 
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1. INTRODUCTION 


The Pluribus* system is a general-purpose multiprocessor 
computer suitable for applications ranging from those normally 
identified with minicomputers to those typically associated with 
larger machines. Pluribus hardware has been designed so as to 
provide a suitable basis for the development of ultra-reliable 


hardware/software systems. 


Pluribus systems contain an arbitrary number of identical 
processors each of which has access both to its own private 
memory and to a common memory accessible by all processors. I/O 
devices which are part of the system can be controlled by any 


processor. The number of processors, size of common memory, and 
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amount of I/O gear on a Pluribus system can be quite large. 


The Pluribus system achieves modularity and reliability by 
making all the processors equivalent. Any processor can perform 
any system task or control any device. Since each subsystem of 
the Pluribus system (processor, memory, and I/O) is expandable, 
systems can easily be configured to meet the throughput require- 
ments of a particular job. The scheme for interconnecting system 
components is also modular; hence, interconnection costs vary 


smoothly with system size. 


The Pluribus system was originally developed to serve as a 
modular reliable packet-switching node for the ARPA Network [1]. 
A node consisting of a 13-processor system is currently operational. 
The Pluribus approach is appropriate, however, for many other 
applications where reliability, modularity, or large logical com- 


puting power is required. 


Trademark of Bolt Beranek and Newman Inc. (BBN) 
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This handbook will describe the structure and operation 
of the Pluribus system. It will emphasize utilization of the 
Pluribus architecture in the manner for which it was originally 
designed, although additional possibilities will become clear 
as the discussion progresses. The handbook is oriented both to 
the programmer who will use it as a basic reference document and 
to the system designer who will have to determine if the Pluribus 
is appropriate for his particular application. Section 2 
presents an overview of the Pluribus architecture. Section 3 
contains a brief description of the processor. Sections 4, 5, 
and 6 present the basic information concerning the Pluribus system; 
addressing, programming, and device handling. Section 7 discusses 
reliability machanisms in the Pluribus system, both hardware and 
software, in detail. Sections 8 and 9 discuss the Infibusses 
and bus couplers. Finally, Section ‘10 describes many device 
dependent features and bits and will be useful most likely for 


reference purposes only. 


The Pluribus is constructed out of components manufactured 
both by BBN and by Lockheed Electronics Company, Inc. (LEC). The 
BBN-produced hardware is described in detail in this document. 
LEC hardware from the SUE minicomputer line is discussed only in 
sufficient detail to make the description of the Pluribus co- 
herent. More complete information on the SUE components can be 
found in the Lockheed product literature [2,3,4,5]. 
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ase PLURIBUS SYSTEM STRUCTURE 


A Pluribus system consists of a number of components 
(processors, memory modules, and I/O devices), a number of 
busses over which these components communicate, and a number 
of bus couplers which provide the mechanism for interconnecting 
the individual busses. Within this framework a wide variety 
of systems can be configured ranging from small single bus 
systems to large multi-bus systems with tens of processors, up 
to 1024K bytes of main memory, and a large assortment of I/O 
gear. The subsequent discussion will focus on a medium sized 
Pluribus configuration. Very small and very large systems. 


both involve additional considerations not discussed in detail 
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here. 


The basic skeletal unit of the Pluribus system is the SUE 
Infibus* onto which all BBN and LEC devices are connected. The 
Infibus not only serves as a chassis into which device cards are 
plugged but also provides a means for communication among all 
attached devices. In general, a single Infibus can have an assort- 
ment of cards on it: processors, memories, or I/O devices. 
However, only one device can be in control of the Infibus at any 
given instant. An -Infibus arbiter (Bus. Control Unit Card--or BCU) 
which must be present on the Infibus guarantees that this is the 
case. The total number of components which can be plugged onto a 
Single Infibus is dependent on the number of slots available for 
cards and the type of power supply used (see section 8.). 
Throughout the remainder of this document the word "bus" will be 


used as a shorthand for Infibus. 


*Trademark of Lockheed Electronics Company, Inc. 
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A small Pluribus system (even a single processor system) 
can be built using only a single bus. For many applications, 
however, the bandwidth capability and/or card capacity of a 
single bus is exceeded and a multi-bus structure is PeQUIred< 
In addition, applications which take advantage of the full capa- 
bilities of the Pluribus hardware for bandwidth and reliability 


will require multi-bus configurations. 


With more than one bus, the question becomes how to assign 
processors, memory, and I/O devices to the individual busses and 
how to connect them together. A typical configuration is illus- 
trated in Figure 1. Lines between busses represent bus couplers. 
Typically, busses in a Pluribus system are configured as one of 
three types: processor busses, memory busses, or I/O: busses. 
Processor busses support processors and private memory associated 
with each of the processors. Up to four processors (numbered 0-3) 
can logically be put on a single bus although contention for the 
bus is likely to reduce the effective processor bandwidth. In the 
ARPA Network application, for example, four processors with con- 
tention produce the same computational capacity as would three 
processors if there were no interference among the processors (i.e., 
if the processors were actually independent). Although the con- 
tention is application-dependent, Pluribus systems will generally 
be configured with one, two, or three processors per processor bus. 
Two processors are indicated for the system illustrated in Figure 
1. The other components normally residing on the processor bus 
are the processor private memories. These memories will contain 
the "hot code" (i.e., those routines most frequently referenced) 
so as to reduce competition for the pool of shared (cammon) 
memory, and other code which is important to protect by 


removing it from shared memory. One useful technique 
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is for all private memories to contain identical copies of the 

same code. Much of the system reliability software will be held 
in the private memories to guarantee that redundant copies exist 
in case of any memory failure. The maximum amount of private 

memory addressable by each processor is 16K bytes. Not shown in 
Figure 1 but existing on every bus is the BCU (bus arbiter) card. 
In certain cases it may be desirable to have some I/O devices on 
the processor bus, but this will be the exception rather than. the 


rule and is discussed further in section 6.4. 


Memory busses contain common memory shared by all processors. 
Up to 1008K bytes of common memory can be added in 8K or 16K byte 
increments. The common memory will typically contain code which 
is referenced less frequently than the “hot. code". Generally, 
Shared data structures, variables, and buffers will also be held 


in common memory. 


The configuration of common memory, that is, the assignment 
of memory modules to memory busses, depends on considerations of 
reliability and memory contention. For both reasons it is desirable 
to have multiple memory modules on a bus, multiple busses, and 
redundant copies of code and data structures. The details are 
application dependent and relate to the cost/performance (relia- 
bility) trade-offs which the system designer must consider. For 
reliable operation at least two memory busses, two processor 


busses, and two I/O busses will be required. 


The I/O busses contain I/O devices and the Pseudo Interrupt 
Device (PID) central to the Pluribus system operation. The PID 
keeps in hardware a list of what to do next. A number can be 
written to the PID at any time and it will be remembered. When 
read, the PID returns (and deletes) the highest number it has 
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stored. By coding the numbers to represent tasks, and keeping the 
parameters of the tasks in memory, a processor can access the PID 
at the end of each task and determine very rapidly which task to do 
next. This approach is an important departure from the use of 
conventional interrupts and avoids the costs associated with saving 
and restoring machine state.* Further, this approach neatly side- 
steps the problem of routing interrupts to the proper processor. 


More detail on the use of the PID is given in section 5. 


There can be no more than four PIDs in a Pluribus system. 
Even though some I/O busses may conceivably not contain a PID, or 


a buS may contain more than one, the usual configuration is one 
PID per I/O bus. 
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In a Pluribus system, processor, memory, and I/O busses are 
eonnected by devices called bus couplers. The different types of 
bus couplers required to connect different bus pair types together 
are discussed in more detail in section 9. For moderate sized 
systems, there will generally be a bus coupler between any two 
busses between which communication is required. Usually this 
implies a coupler from each bus to all busses of other types. 
Thus the total number of bus couplers for such a Pluribus system 
with P processor busses, M memory busses, and I I/O busses is 
Pe Mek + Pel, Por smaller systems 10-is possible for one or 
more Infibusses to serve as combined memory and I/O busses, 
reducing the number of required bus couplers. For applications 
requiring large numbers of components (processors, memories and 
T/O devices) it will be possible to reduce the required number of 


bus couplers by building hierarchical Pluribus systems where 


*Although not used within application software, conventional 
interrupts can result from errors and are used for special purposes. 


See section 3.3. 
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busses are not completely connected. A more detailed discussion 
of the issues and procedures for configuring a Pluribus even 


can be found in a separate document (6. 


several distinct processor models are available. The SUE 
is a relatively slow and inexpensive processor. Typical memory- 
to-register instructions have execution times on the order of 4 
microseconds. For a given application, the required processor 
power can be attained by using as many processors as are necessary. 
This approach to generating high throughput systems has the 
advantage of permitting extreme modularity and high reliability 


as well as graceful degradation. 
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3% PROCESSORS 
3.1] Instruction Set and Format Summary 


Lockheed SUE processors are used as processor components 
for Pluribus systems. The basic processor is a microprogrammed 
general purpose 16-bit minicomputer with 8 general registers (one 
of these registers is the program counter), and a status and a 
control register. These registers (general purpose, status, and 
control) may be accessed externally by other devices via the 
Infibus. In a multiprocessor this allows one processor to stop 
another, examine and change its registers, and restart it. There 
are 8 general instruction classes: MOVE, ADD, SUBTRACT, INCLUSIVE 
OR, EXCLUSIVE OR, AND, COMPARE, and TEST. Each of these instruc- 


tions can use a variety of addressing modes including register-to- 
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register, memory-to-register, register-to-memory, indexed, in- 
direct... and auto-indexed:. Also available are rotate, shift, 
eonditional branch, unconditional branch, and subroutine caliins 
LNSUPUCTIONG.. The. conditions tested. for branching are bits in 

the status register of the SUE. There are tests for result being 
Zero, result being negative, carry on. last aratvhmetic instruction, 
register value odd, overflow, value greater-than on last comparison, 
value equal on last comparison, and loop completion. The branch 
Gan. -OCCUr on. elther valve TRUE or FALSE. Instructions: are elther 
one or two words long. The processor also contains 3 programmable 
flags, contained in the status register, that can be manipulated 
directly by instructions. The sun processor recornizes. and 
generates 16-bit addresses. In addition it contains a 2-bit KEY 
register which is settable by the SKEY instruction in The pro- 
cessor: ‘The contents of Uhis register are appended to the most 
significant end of the 16-bit address to generate an 18-bit address. 


Every memory access by a processor has these two bits appended. 
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Certain of these 18-bit addresses are mapped into 20-bit system 
addresses as described in section 4. The processor operates on 
either 16-bit words or 8-bit bytes of data. Bit @ is identified 
with the low order bit and bit 15 with the high order bit. Details 
of the SUE instruction set and the various processor types may be 


found in reference 4. 


S22 PROCESSOR STATES 

SUE processors can be in one of three states: halted, 
running, or idle. Transitions between these states may be ef- 
fected either by the processor itself or by external manipulation. 


The implications of each of these three states are as follows: 


Halt: No instructions executed, interrupts disabled, 


registers externally accessible 


Run: Instructions executed, interrupts enabled, 
registers not externally accessible except 


control register. 


idle: No instructions executed, interrupts enabled, 
registers not externally accessible except 


control register. 


External references to registers which are not accessible 

will result in a QUIT, as described in section 3.3. The Idle 
state is entered from the running state by executing a WALT 
instruction; the HALT state, by a HALT instruction. The Run. 
state is entered from the Idle state by the occurrence of an 
interrupt. External manipulation of these states operates as 
follows: 

The processor control register is the only register accessible 
while the processor is running. To halt a processor, a one is 


written to its control register. The processor will normally halt 


10 
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when the instruction it is currently executing is completed However, 
if the control register is read between the time that a zero 

is written to the control register and the time that the pro- 

cessor completes its current instruction, the halt signal will 

be lost. Hence, some algorithm such as the following should 


be used to guarantee that a processor does in fact halt: 


L: Write 1 to the processor control register. 
Read some other processor register. 
If QUIT results go to L (see section 3.3). 
At this point the processor is halted. 
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To start a processor, one must initialize all important 
registers to needed values and store the number two into the 
control register. This is done as follows: First it must 
be assured ‘that. the processor is halted, ‘The program counter 
is initialized to the address of the program to be executed. 
The general registers are loaded with any values to be passed 
to the program. The status register is initialized to specify 
the enabled interrupt levels, initial status flags, and pro- 
grammable flags. Bit 11 (hexadecimal constant @8@2) should 
be set to activate the processor. Finally, writing a 2 to 
the control register starts the processor. An additional 
feature of the SUE processor is the ability to single step 
through an instruction sequence. The procedure fcr doing 
this is identical to that of starting a processor except 


that a 3 is written to the control register rather than a 2. 


1] 
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3.3 QUIT Handling 


Processors requesting access to memory locations or I/O 
registers do so by directly or indirectly placing the desired 
address on the appropriate bus along with the required operation 
(e.g. read, write) and any data required (for a write operation). 
When the requested operation is complete, the processor will | 
receive a signal called DONE. If no device on the destination 
bus recognizes the address provided or if the device recognizing 
the address malfunctions, no DONE signal will be returned to 
the processor. Instead, after a fixed period of time, the bus 
arbiter on the requesting processor bus will send a QUIT signal. 
to the processor, causing a conventional interrupt. (An equi- 


valent thing happens to requests by I/O devices from I/O busses.) 


In many applications a programmer will want to take some 
action based on the presence or absence of QUIT interrupts. In 
the ARPA Network application, for example, a device discovery 
routine in the reliability software searches system address space 
and determines if known devices have disappeared or new ones 
appeared by attempting to.read the devices! registers and 
checking for resulting QUITS. To provide this mechanism the 
interrupt level routine which responds to QUITs should be written 


so that control will be passed back to the application program 
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if the application programmer has indicated that he wishes this 
to happen. He indicates this wish by surrounding the instruc- 
tion which potentially causes the QUIT by an "unusual pattern" 
of other instructions. For example, if a programmer wants to 
check for a QUIT occurring when location ABC is referenced he 


might write the following: 


LDA A2, ABC 

NOP 

BR - + 4 
i QUIT BRANCH ADDRESS 


IT no QUIET. program continues here. 
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The QUIT interrupt service, upon receiving control, would check 
to. see 1f the two. instructions. following the one whicn caused 
the QUIT were NOP and BR. +4. If they were, it would simply 
store the two bytes starting at location L (the address of the 
instruction causing the QUIT plus 8) in the program counter and 
dismiss the interrupt. If the two instructions at L-4 and 

L-2 do not match the NOP BR. +4 pattern, the interrupt service 
routine would take its usual action in handling the QUIT. Of 
course, references to ABC which do not cause QUITS cause 


execution to continue at Lte as indicated. 


es 
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4. ADDRESSING 


A typical Pluribus configuration incorporating both 
private memory associated with each processor and a pool of 
common memory shared among all processors has been presented 

in section 2. In this section the Pluribus address structure 

is described in more detail. The application of this address 
structure to Pluribus program structures will be discussed in 
section 5. 


In Pluribus systems all devices communicate with one 
another by writing into or reading from addresses. These ad- 
dresses may be memory locations, locations for controlling or 
interrogating I/O devices, or they may have some other special 
function. In any case it is important to understand two things; 
first, how addresses are generated and routed through the system 


and second, what things are referenced by what addresses. 


Addresses are generated by active devices, that is devices 
wanting to read or write some location. This includes both 


. processors and.I/O devices. Consider first a processor. 


As indicated in Figure 3(a), SUE processors normally 
generate 16-bit addresses. With the addition of the 2-bit KEY 
register in the SUE, however, the Pluribus processors actually 
generate 18-bit addresses which are put on the processor bus. 
The KEY register can be changed under program control by 
execution of the SKEY instruction. For the class of applica- 
tions being considered in this document, however, each processor 
on a bus will initially set its KEY register to a distinct 


value indicating its physical processor number and will not 


14 


Report No. 2930 Bolt Beranek and Newman Inc. 


PRIVATE MEM 
FOR PROC | 


O 


PROCESSOR. ADDRE 
SS I ESES FFF 


SOOO-3FFF 
4900-5FFF 


6GOOD-TFFF ) RELATIVE TO 
80QQ0- SFFF ; MAPS 1,2,3 


AGQ@-BFFF ) RESPECTIVELY PIODYP 
C@O@- FBFF 
FCOQ-FFFF 


COMMON ADDRESS 


& 
2 
ut 

ok 
_ 

O 

VY) 

® 
Q 


PROCESSOR 
REGISTERS 
AND LOCAL 
1/0 SPACE 
FC 
990 SYSTEM 
I/O 
FFBFF SPACE 
Figure 2 Processor Address Space 


15 


Report No. 2930 . : Bolt Beranek and Newman Inc. 


KEY (2) | 18 BIT ADDRESS ON PROCESSOR BUS 
ae 2) Glee ela, 
PROCESSOR ADDRESS (16) 
ah 

(a) 
I8 BIT ADDRESS ON PROCESSOR BUS 14BIT ADDRESS RECOGNIZED BY 


PROCESSOR -XYS PRIVATE MEMORY 


MAP REGISTER (7) 20 BIT ADDRESS PUT ON MEMORY BUS 
ee | enn 
i8 BIT ADDRESS ON PROCESSOR BUS | 


Xyrst] 
rs=Gl,1D 


st selects map register (c) 
20 BIT ADDRESS PUT ON 1/0 BUS 


I8 BIT ADDRESS ON PROCESSOR BUS 


tuvw ¢ lill 


(d) 


I8 BIT ADDRESS ON PROCESSOR BUS 


777 7 | 


ADDRESSES OF THIS FORM REFERENCE MAP REGISTERS, PROCESSOR BUS 1/0 
- DEVICES, AUTO LOAD ROM, AND PROCESSOR AND CONSOLE REGISTERS 


(e) 


Figure 3 Address Mappings 


16 


Report No. 2930 Bolt Beranek and Newman Inc. 


normally change this setting. 


The KEY bits thus serve to differentiate the address spaces 
of the various (up to four) processors on the bus. The right- 
most bit of the 16 left to each processor is used to select left 
or right byte in byte mode instructions, allowing gid = 32K 
addressable words. In order to allow larger common memory and 
I/O space than this, provision has been made for mapping portions 
of this 32K processor address space onto a 512K word system ad- 


dress space. 


The addresses shownatthe left in Figure 2 are the 16-bit 


addresses generated by a typical processor (processor j). The 
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manner in which the address space is accessed by any processor 


generated address depends on the range in which that address lies. 
The four types of access will be discussed individually below. 

With all 4 types of access being discussed, the 18-bit address 

is simply put on the processor bus. Devices (memory, bus couplers, 
and I/O gear) able to recognize that address will respond and 


all others will ignore the address. 
4.1 References to Private Memory (QPQO-3FFF): 


Any processor generated address in this range refers toa 
location in the private memory associated with processor j on 
its processor bus. Up to 16K bytes of private memory can exist. 
As shown in Figure 3 (b), the high order two bits of the 18-bit 
address are used to select the proper memory module and the low 


order 14 bits select the location within the private memory. 
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4.2 References to Common Memory (4P9P-BFFF): 


Any processor generated address in this range refers to a 
location in common memory, that is, memory on one of the memory 
busses. There are 4 distinct sub-ranges within this range, each 
associated with a distinct hardware mapping register. This 
association is indicated in Figure 2. Each map register allows 
a contiguous set of 8K bytes of commom memory locations to be 
referenced. A separate set of 4 map registers is associated with 
each processor on a processor bus. The map registers are physi- 


cally located in the bus couplers (see section 9.1.) 


Figures 2 and 3(c) illustrate how this mapping into common 
memory is accomplished. An address in the range 4QQ00-5SFFF 
implicitly refers to map register @. A 20-bit system address is 
developed at the processor end of the coupler by appending 7 bits 
from the map register to the low order 13-bits of the 18-bit 
address on the processor bus. This address is then forwarded to 


the common memory bus with the access request. 


Addresses in the range 6000-7FFF, 8000-9FFF, and AOOO-BFFF 
implicitly refer to map registers 1, 2, and 3 respectively and 
the identical type of mapping occurs when these sub-ranges are 
referenced. The only special feature in the way that the four 
maps work is related to memory read operations via map register 
3 which are transformed into read-modify-write accesses to 
common memory where the data rewritten is always zero. This 
allows the implementation of multiprocessor locks in the Pluribus 
system. More detail on the use of this feature is discussed in 


section 5.3 where Pluribus program structures are considered. 
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One final complication arises from the fact that a few of 
the first addresses on every memory and I/O bus are allocated for 
accessing the bus coupler control registers. The amount of this 
allocation depends on the number of couplers connected to the bus. 
In general, the memory words at these addresses should not be 
used. For more detail on the bus coupler control registers see 


section 9.2. 


4.3 References to System I/0 Space (CPPP-FBFF): 


Any processor generated address in this range refers to a 
location in system I/O Space. In general, each Pluribus system 


device on an I/O bus appears to the processor as a set of 8 
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contiguous registers (locations) in system I/O space. This block 


of registers is referred to as the device register block. A 
processor can activate a device by writing commands or data to 
certain (device dependent) addresses within the device register 
block. <A processor can interrogate a device by reading data or 
status registers within the 16 byte device register block. More 
detail about the allocation. of system 1/70 space vo multiple 170 
busses and about the internal structure of the device register 
block can be found in section 6 where device handling and I/O is 


discussed further. 


As indicated in Figure 3(d), the way that system I/O space 
addresses are developed is by appending four ones to the low-order 
16 bits of the 18-bit address on the processor bus. This is done 
automatically as is the appending of the map registers (discussed 


above) by hardware in the bus couplers. 
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4.4 References to Maps, Processor Registers, and Local I/0 Space 
(FCOO-FFFF): | 


Any processor generated address in the range FCOO-FFFF 
(see Figure 3(e)) refers either to the map registers for that 
processor (in all the bus couplers attached to the processor 
bus) or refers to a part of the address space shared by all 
processors on a processor bus. The map registers must be ad- 
dressable, of course, so that a processor can dynamically modify 
the portions of the potentially large common memory to which it 
has access. Map registers 9, 1, 2, and 3 can be referenced via 
addresses FCOO, FCO2, FCOH4, and FCO6 respectively. 


The local (to the processor bus) shared address space is 
assigned as shown in Figure 4. In general, I/O devices will be 
attached to an I/O Infibus in a Pluribus system. In some cases, 
however, it may be desirable or necessary to connect I/O devices 
directly on a processor bus. Addresses in the range FCO8 to 
FDFF will be used in such a configuration to refer to the device 
registers, similar to the way that the device register block is 
used for referencing devices on the I/O Infibus. The auto load 
ROM is an optional hardware device attached to the processor bus 
which contains a program that when executed will cause a processor 
to load memory from a paper tape reader on the processor bus. 

The registers of all the processors and the processor bus console 
are accessible at addresses above FFOO. A processor should be 
halted before an attempt to read any of its registers occurs. 


Halting a processor is described in section 3. 
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Des PLURIBUS PROGRAM STRUCTURE 


In most current computer systems a hardware priority 
interrupt mechanism is used to inform the program of the oc- 
currence of asynchronous external events. Since Pluribus systems 
_ do not generally use interrupts for this purpose, Pluribus pro- 
grams tend to be structured differently from programs developed 
for conventional machines. The fact that Pluribus programs are 
designed to operate in a multiprocessor environment imposes ad- 
ditional constraints on the program structure. This section 
presents some of the issues and programming techniques which we 


believe are useful in developing Pluribus programs. 
5.1 Basic Control Structure: 


Before giving an example of a typical Pluribus program 
control structure, the basic operation of a PID will be reviewed 
(more detail on the PID can be found in section. 10.1). The PID 
is a priority ordered memory device. It has a read address and 
a write address. When an even 8-bit number is written to a PID, 
the number is stored. When a PID is read, the largest 8-bit 
number stored in the PID will be returned and the number deleted 
from the PID. If nothing has been written to the PID, the read 
will return a value of zero. Numbers may be written to the PID 
both by hardware I/O devices and by software. Processors poll 
the PID for tasks to be executed. AS a simple example of a 
Pluribus control structure, consider a system consisting of a 
number of tasks which service a set of 1/0 Gevices. The fol- 
lowing assembly language code could provide the framework for 


the required program. 
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TASKDISPATCHTABLE: MAINLOOP, TASK1, TASK 2, ..., TASKN 


MAINLOOP: LDA Al, PIDREADADDRESS 
JMP @ TASKDISPATCHTABLE (Al) 


TASKi: 
JMP MAINLOOP 


The main loop of the program simply reads the PID and jumps to 
the appropriate task indirectly through TASKDISPATCHTABLE (i) 
where i is the value obtained by reading the PID. At the end 
of any task (e.g. TASKi), a jump to the main loop returns the 
processor to look for the next task to perform. If there is 
nothing in the PID, zero is returned and the processor simply 
cycles at MAINLOOP. Note that it is useful to have the PID 
store even numbers only since the number retrieved will be used 


as an index into a table with two-byte entries. 


To allow tasks to be initiated by the software (e.g. TASKi 
to be Initiated by TASK] ), the Tollowing type of structure 


would be used: 
TASK I: ) 


LDA Al, TASKiPIDLEVEL 
STA Al, PIDWRITEADDRESS 


JMP MAINLOOP 
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9.2 System Response Time and Strips: 


As indicated in the above example, the way that I/0 
devices obtain service in a Pluribus system is to write the 
priority level of their service routine to the PID when they 
need attention, and wait for some processor to return to the 
main loop and pick up the associated task. Since the time 
that a device can wait for service before losing data may be 
critical, it is essential to configure systems and design soft- 


ware so that response time requirements can be met. 


The two main factors which influence the rate at which a 
Pluribus system can respond to high priority external events are 
the total number of processors in the system and the duration 
of task servicing instruction sequences. For example, ina 
Single processor system where the tasks are all of the form 
illustrated by the two previous examples, if the longest task 
execution time were T milliseconds, the maximum time which it 
could take to respond to an external event (i.e., notice that 
it had occurred) would also be T milliseconds. This worst case 
would happen only when the event occurred just after the single 
“processor had picked up the longest task to run. Since in @ 
Pluribus system there are no interrupts, the entire task cur- 
rently being executed runs to completion before there 1s a 
reaction to the event (even though it may be of higher priority 


than the task currently being run). 


In the multiprocessor case, things are slightly more com- 
plicated. Considering the worst case response time as above, if 
the ordered task execution times are Ty (smallest), Ts» T3. 

Tn (largest) and there are P processors, the maximum time to 
respond to an external event assuming n>p will be between 


om, 


between Th and T depending on the number of incarnations of 
Nn : 
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a particular task which can exist simultaneously. Of course, 

the probability of such worst response times may be exceedingly 
small if the large tasks are run less frequently than the smaller 
tasks. 


Typical (average) rather than worst case response times 
will depend on three factors: (1) average task execution time, 


(2) number of processors P, and (3) average number of tasks, 


Na, queued on the PID. If the average task execution time is 
T.. and a the typical time taken to service a high priority 
event will be T,,/2. If P>N, then there will usually be an idle 


processor which will immediately react to the external event and 


average response time will be essentially zero. 


In general, the application will dictate where strict 
real-time response must be guaranteed or if more flexible system 
response characteristics are adequate. If strict real-time 
response is required, then some program structure which permits 
both logical tasks of arbitrary length and fast response to 
critical external events may be required. To accomplish this, 
Pluribus program tasks can be partitioned into code segments 
referred to as strips. A strip is simply a sequence of instruc- 
tions within a task. A task can give up control of its proces- 
sor at the end of each strip so that any higher priority tasks 
may be run. Of course, if the task is incomplete at the end of 
a strip, the task queues itself on the PID for further execution 
before yielding its processor. The idea is illustrated by the 


example below where TASKk is broken down into two strips. 
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DISPATCHk: Kil /INITIALIZE TO THIS VALUE. 


TASKk: JMP @ DISPATCHk 


Kl: 
LDA A2, = K2. 
STA A2, DISPATCHk 
LDA Al, TASKPIDLEVEL Strip 1 
STA Al, PIDWRITEADDRESS 
J MP MAINLOOP 
Ke: 


LDA A2, = Kil 
STA A2, DISPATCHk | Strip 2 
JMP | MAINLOOP 


The first instruction of TASKk is a dispatch to the segment 
(strip) of the code to be executed. This dispatch is initialized 
to Kl so when TASKk is first initiated, execution will begin at 
Kl. At the end of strip 1, the task stores a new dispatch ad— 
dress (K2) in the subtask dispatch location, DISPATCHk , writes 
its own PID level back into the PID and gives up the processor. 
The next time this PID level is serviced, the task will be re- 
sumed in strip 2 starting at K2. At the end of Strip 2, the 
subtask dispatch location is restored so that strip 1 will be 
executed the next time that TASKk is activated. It must be kept 
in mind that a task writing its own level to the PID will pre- 
vent the processor which is executing the task from picking up a 
waiting task with lower priority. In certain situations it may 
be desirable for a task to yield the processor and aiso 
a specified period prior to getting rewritten to the PID. This 
can be accomplished by the task setting a software timer which 
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gets counted down by a periodic clock routine. When the timer 
reaches zero, the clock routine can write the sleeping task's 
level to the PID. The 5 instructions at the end of strip 1 in 
the above example might, therefore, be replaced by the following: 


LDA 22. = Ke 

STA A2, DISPATCHk 
LDA A2, SLEEPTIME 
STA A2, TIMERk 
JMP MAINLOOP 


Then after TIMERk has been counted down, the timer routine will 


execute the instruction: 


G 
ao) 
per) 

Qo. 
he 

OQ. 

) 

® 
a) 


LDA Al, TASKKPIDLEVEL 
STA Al, PIDWRITEADDRESS 


The decision of precisely where to segment a task into 
strips is somewhat arbitrary; the main rule is that the strips 
must be short enough so that the proper response characteristics 
can be guaranteed. In the ARPA Network application of the 
Pluribus, for example, it turned out that the proper typical 
strip size was on the order of 100 instructions (although a few 
infrequently run ones are much longer). As a rule of thumb, it 
will generally be sufficient to segment a task into strips as- 


suming each instruction takes 4 usee for execution. 


Two other related practical issues relevant to strip size 
selections are convenience and overhead. In general, tasks should 
be broken into strips at convenient points in the code; that is, 
points at which little information (e.g. in the registers) needs 
to be preserved. It may occasionally be desirable to have strips 


somewhat smaller or larger than the nominal size so that such a 
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partitioning will be-possible. Data which must be saved and re- 
stored across strip boundries adds to the already existing over- 
head associated with breaking the code into strips. In many 
applications it is likely that little or no breaking of tasks 
into strips will be required. In the ARPA Network application, 
for example, multi-strip logical tasks are the exception rather 
than the rule. 


The fractional overhead associated with breaking a task into 
strips depends directly on the strip size since the number of 
instructions required for strip switching is essentially fixed. 
For example, in TASKk presented above 8 overhead instructions are 
associated with switching from one strip to another (6 in TASKk 
and 2 in the main loop). If the strip size were 100 instructions 
as is typically the case in the ARPANET application, then the 
processor overhead due to using strips would be 8%. In applica- 
tions where: larger .strips are acceptabie, of course, the .over= 
head will be even smaller. Experience with a number of Pluribus 
System applications has indicated that the processor overhead 
and programmer effort associated with breaking tasks into strips 
is not a serious problem and is a relatively small price to pay 
for the increased reliability and performance of the novel 


Pluribus architecture. 
5.3 Shared Data Structures, Shared Code, and Locks 


In a multiprocessor care must be exercised when a piece of 
data may be referenced (read and/or written) simultaneously by 
more than one processor. In this context, "simultaneously" 
means that a process running on one processor desires access to 
the data while another process running on a second processor 
already has access to the data. Consider, for example, two 


processors that are concurrently executing processes which 
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obtain buffers from a common free storage list. If some interlock 
is not used, it would be possible for both processors to get the 
same buffer since the second processor could access the list after 
the first processor had accessed it but before the pointer was 
updated. 

To avoid this and a multitude of similar situations involving 
Shared resources, a lock mechanism is typically used in programs 
for multiprocessors. Before a shared resource is accessed by a 
process, a logical lock is set. All processes determine if the 
lock is set prior to accessing the resource, and if so, then the 
process will wait. Only one process can, therefore, have access 


to the shared resource at any one time. 
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To be effective it must be possible to test and set a lock in 


a Single operation. A typical implementation provides the 

ability to read, test, and provisionally modify a memory location 
in a single uninterruptable operation. In Pluribus systems this 
feature is provided by turning memory reads through map register 3 
into read-modify-write accesses where the data rewritten is all 


Zeros. 


To implement a lock on a shared resource one simply assigns a 
location (LOCKVAR), addressed through map register 3, to the 
lock. The resource is unlocked if the lock is non-zero and 
locked otherwise. A segment of code which accesses a locked 


resource might look as follows: 


is LDA A2, LOCKVAR (Lock and continue) or (WAIT) 
BZ L 


access to shared resource 


bie Sm PC. GOCKVAR 7 Unlock Lock 
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If a processor falls through the loop at L, the resource was 
unlocked but is now locked by the process running on this proces- 
sor. If a processor loops at L, then the shared resource is in 
use and the processor waits until the lock is released. To unlock 
the resource at Ll any non-zero quantity could have been stored in 
LOCKVAR. The current program counter (PC = general register 0) 
contains one such value which has the additional advantage of 
leaving a partial trace of the program execution in the lock 
registers. This trace may be helpful for debugging purposes. 


When a process encounters a locked resource, it may take one 
of two actions. As in the above example, it can remain in a tight 
loop checking the lock until it is unlocked. This type of waiting 
will be called busy waiting since the processor running the pro- 
cess remains occupied while waiting. Alternatively, a form of 
non-busy waiting may be implemented where the process may either 
write itself to. the PID or set a timer so that a clock routine 
will later write it to the PID as described cariier. *In.celther 
case the processor then is free to seek other tasks while waiting. 
The busy form of a lock is appropriate in situations where the 
resource will be locked for only a short period. An example of 
this is the free buffer list accessing mentioned at the begin- 
ning of the discussion on locks. The lock implementation which 
dispatches the processor to do other useful work will be more 
suitable in situations where the shared resource is likely to 
remain locked for a relatively long time. A paper tape reader 


shared by two processors might be such a resource. 


The preceding discussion leaves considerable latitude with 
respect to what should be locked and when. For example, if each 
incarnation of a piece of shared code references a set of shared 


variables, it may be more efficient to associate a single lock 
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with the set of shared variables than a lock with each of the 
individual variables. What needs to be balanced against this goal 
of fewer locks is the desire to keep locked segments short. 

Large locked segments, while reducing the total number of locking/ 
unlocking operations required,will tend to increase overhead due 
to increased busy waiting or processor task switching. This 
overhead can become quite large as the percentage utilization of 
the shared resource increases beyond 60 - 70%. For this reason, 
the system designer must use considerable judgement in deciding 
on the extent of locked segments. In addition, locks should not 
remain locked across strip boundaries. Locked segments should 


also be executed with interrupts disabled so that prompt unlocking 
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of the shared resource iS assured. 


One further consideration is that a processor may fail while 
executing a locked segment. Two problems can arise in this case, 
(1) the locked resource will be unavailable to other tasks and 
(2) if busy waiting is implemented, processors may be executing 
infinite loops. Therefore, a processor should only be allowed to 
wait for the maximum amount of time which the lock can legiti- 
mately be set before deciding that a malfunction has occurred 


and activating a recovery procedure. 


Cooperation with respect to the use of shared variables 
1s required between tasks corresponding to different code 
segments and especially tasks corresponding to different incar- 
nations of the same reentrant code segment. In general, reentrant 
coding is particularly appropriate in a multiprocessor such as the 
Pluribus system. The shared code may exist in common memory 
or multiple copies of the code may exist in the private processor 
memories to reduce contention. In the ARPA Network application, 
for example, shared code is used to transmit data from the IMP to 


each of a number of modems. In this case, the control structure 
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illustrated earlier in this section is modified to look as 
follows: | 


“TASKDISPATCHTABLE: MAINLOOP, TASK1, +++, MODEMOUT, MODEMOUT, -+*TASKN 
CONTROLBLOCKS: @, BLOCK1, *** , MBLOCK1, MBLOCK2, «++ BLOCKN 


MAINLOOP: LDA Al, PIDREADADDRESS 
_ -JMP + @TASKDISPATCHTABLE (Al) 


MODEMOUT: LDA A2, CONTROLBLOCKS (Al) 
LDA A3, MODEMLOCK (A2) | 


STA PC, MODEMLOCK (A2) 
JMP MAINLOOP 


The modem interfaces each write different levels to the PID when 
output of a buffer is complete but all these levels activate the 
same piece of shared code, MODEMOUT. The PID levels are used, 
however, to select the address of a control block which contains 
the variables specific to the modem being serviced. At the start 
of MODEMOUT, an instruction is executed which loads an accumu- 
lator, (A2), with the address of this control block. One of the 
words in this block is a lock used to lock all the other shared 
variables in the block. Thése variables remain locked for the 
duration of the modem output tasks. 


5.4 Using the Map Registers: 


The map registers allow four independent 8K byte segments of 
the common memory to be referenced by each processor. The only 
constraint is that a read done through map register 3 will be a 


read and clear. The other three map registers may be used To 
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point to program or data as required by the application. It is 

possible to have two map registers point to the same segment of 

memory. In the ARPA Network application, for example, map 3 

and one of the other map registers point to a segment containing 
System variables which can be accessed normally or used as lock 


variables. 


In Pluribus systems with small memory configurations little 
Or no map changing may be required. For applications requiring 
large primary memories, map changing will be more frequent. Of 
course, it is desirable to design a system so that as little map 
changing as possible will be required. To change the area of 


common memory addressed through a particular map register, one 
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Simply stores into the map register a constant whose high order 


{ bits are to become the contents of the mao. As already mentioned 
in section 4.4, the four maps have addresses FCO0O, FCO2, FCO4, and 
FCO6. The code which changes a map must not itself be referenced 
through that map. One way to make sure that this does not occur 


is to execute all map changing code out of private memory. 
9.9 Using Multiple PIDs 


Tie PID is: Une- heart. OT the Pluribus. systems. Bssentialiy «als 
task dispatching is done via this device. It is important, therefore, 
that reliability provided by redundancy in the remainder of the Plu- 
ribus system components not be jeopardized by availability of only a 
single PID. 

in &@.mubti=riD system, the PLDs will. themselves be: priorivy 
ordered. Typically, the control program in such a system will read 
the highest priority PID first. If a PID other than the lowest 
pPloricy PTD Pepurnis Zero, tne next Lower priority ip wis 


be read. If all PIDs return zero, the control program simply 
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cycles by reading the highest priority PID again. 


As indicated earlier, a Pluribus system can have up to 4 PIDS, 
one on each of 4 I/O busses. A hardware device on an I/O bus is 
associated with a PID on that bus. Software tasks, on the other 
hand, may write to any of the PIDs in the system. Redundant I/0 
devices will generally be on different I/O busses and associated 
with different PIDs. | 
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6. DEVICE HANDLING AND I/0 


Pluribus systems may be comprised of two types of I/O 
devices, BBN-developed devices and Lockheed-developed devices. 
The primary distinctions between the two are that BBN devices 
interpret 20-bit addresses and use the PID while Lockheed devices 
interpret 16-bit addresses and utilize the standard SUE priority 
interrupt mechanism. Since SUE I/O programming is discussed at 
length in L2], most of this section will be devoted to the 
specifics of programming BBN-developed devices. Special con- 
siderations relevant to the programming of Lockheed I/O devices 


in a Pluribus environment are given at the end of the section. 


6.1 Address Structure 

As shown in Figure 2, system addresses FCO00 to FFBFF are 
reserved for Pluribus system I/O space. The detailed structure 
of this space depends on the allocation of addresses to I/0 
busses. Figure 5 shows one possible allocation of addresses in 
the case of a Pluribus with 2 I/O busses. Possible variations 


on this structure will be indicated later. 


The total system I/O space in Figure 5 is divided into four 
almost equal parts, two of which are assigned to each bus. The 
high address segment for each bus will be referred to as the 
primary I/O space and the low address segment as the auxiliary 
I/O space. Note that the primary address space of bus 1 (from 
address FFOOO to FFBFF) is shorter than the other 3 segments by 
1024 bytes because these 1024 addresses are allocated to 
individual processor maps, registers, and local I/0 space as 
Shown in Figure 2. At the beginning of each primary address 
space are 144 bytes of reserved addresses. These locations are 
associated with the clock (RTC) and PID on the bus (see sections 


10.1 and 10.2), contain the bus coupler (BCM) control registers 


35 


c 
2 
ed 
Qo. 
= 
z) 
” 
@ 
a) 


Report No. 2930 | Bolt Beranek and Newman Inc. 


PID WRITE 


PID READ 


PID CLEAR 


SYSTEM 
CLOCK COUNTER 
CLOCK PID 
LEVELS 


CLOCK READABLE 
REGISTER 1 


CLOCK READABLE 
REGISTER 2 


CLOCK READABLE 
REGISTER 3 


PID & RTC 
BLOCK 


COUPLER 
CONTROL 
REGISTERS 


BC | 
CONTROL 
REGISTER 


BBC 


COUPLING | Fegaa miNpom 


REGISTERS 


DEVICE 
BLOCKS 


(8 WORDS 
EACH ) 


FE@6E | BBC MAP 


FEFFF | 


Figure 5 System I/0 Space 
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(see section 9.2), and provide mapping for backwards bus coupling 


(see section 7.1.3) using this bus. 


The remainder of the system I/O space is divided into 16-byte 
blocks where each block is associated with an I/O device (other 
than the clock and PID) attached to the bus. These blocks are 
called device register blocks. <A processor activates an I/O 
device by writing to a certain address within the device register 
block. A processor can interrogate a device by reading the con- 
tents of status registers contained in this block. More detail 
on the structure of device register blocks is given below and is 
also contained in section 10. where individual I/O devices are 


discussed. 


Variation of the structure shown in Figure 5 depends on the 
number of I/O busses and the allocation of system I/O addresses 
among them. This allocation is determined by switches on the bus 
couplers (see section 9.2). Figure 6 indicates allocations of 
system I/O space for 1, 2, 3, and 4 I/O busses. Only the primary 
I/O space allocations are shown; the auxiliary allocations are 
identical to these except that the highest address segment of 
auxiliary is the same size as the rest of the segments, that is, 
it is not reduced in size by 1024 bytes. The low 144 bytes of 
each primary segment is reserved on each bus as indicated in 
Figure 5. While other allocations are possible, the ones shown 
in Figure 6 constitute all of the reasonable ones. Switch 
settings resulting in non-contiguous primary and auxiliary seg- 
ments for individual busses, while possible, are not considered 


here. 
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BUS @ 


FFBFF 


(a) ONE BUS 


(b) TWO BUSSES 


(c) THREE BUSSES 


(d) FOUR BUSSES 


Figure 6 Allocations of Primary System I/0 Space 
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6.2 Programming BBN DMA (Direct Memory Access) I/0 Devices 


BBN DMA (Direct Memory Access) devices provide a means for 
the automatic transfer of blocks of data to (from) memory from (to) 
I/O devices on the I/O busses. While the DMA hardware and its 
associated device interface are on Separate cards, from the 


programmer's viewpoint they may be thought of as a single unit. 


In general, each data transfer will involve sending or re- 
ceiving a number of data buffers. Each data buffer will consist 
of an integral number of words, i.e., an even number of bytes. 
For each direction of data flow (read, write) there are three 
main registers used by the programmer to control I/O operations; 


the begin memory (buffer) address register, the end memory (buffer) 
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address register, and the status register. These registers are 
contained in the 16-byte device register blocks. The structure 
of the device register blocks for BBN DMA devices is shown in 


Pigure 7. Each of these registers is described in detail below. 


DEVICE TYPE - The high order byte contains a number indicating 
the type of device interface involved (e.g. modem, host, etc.). 
This number is fixed by hardware in the device interface associa- 
ted with the DMA. In general, the low order byte contains the 
value set in the device number switches in the device interface. 
The device type register is readable; writing to it will have no 
effect. 


RECEIVE/TRANSMIT BEGIN ADDRESS - These registers contain the 

high order 16 bits of the 20-bit system address specifying the 
first. ecation of the bulber to be. wead or written: Bits 13 

of the 20-bit starting address are contained in the receive or 
transmit status register (see below). Bit O of the 20-bit system 


address is always @. The beginning address registers may be 
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either read or written. If read, the result returned is simply 
zero. Normally when writing into this location, no data trans- 
mission will be in progress in the direction corresponding to 

the register written (receive or transmit). The device will 
simply be initialized to transfer a buffer; actual data transfer 
does not commence until the buffer end register is written. 

If a transfer is in progress when the location is written, the 
transfer is aborted, the error bit (in the end address register 

- see below) is set, the PID is written, and the corresponding 
half (receive or transmit) of the device is initialized for trans- 


mission of a new buffer. 


RECEIVE/TRANSMIT END ADDRESS - These registers may be read or 
written. Normally, bits 0-12 of these registers will be written 
with the low order 13 bits of the address of the end of the 
buffer. (it O is actually ignored and assumed to be zero.) 
Writing to this address initiates the data transfer. After the 
data transfer has ended, these registers can be read to determine 
information concerning the way that the transfer completed. 

Bit 15, if set, indicates that no error has been detected and 
that this was the last buffer of the transfer. (Bit 15 will be 
set when the last buffer is transmitted correctly.) Bit O serves 
as an error bit and will be set if: (1) the device was reini- 
tialized during the previous transmission (see above), (2) a QUIT 
occurred during transmission of the previous buffer, (3) the 
device is currently active (see RECEIVE/TRANSMIT STATUS below) or 
(4) the device itself is reporting an error. Bits 1-12 of the 
end address register indicate the address, modulo ee of the 
last word actually transferred. The top 7 bits of the DMA pointer 
into the buffer come from the begin address (see above) and never 
change. Therefore, the buffer will "wrap around" on 8K byte 


boundaries in memory. 
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RECEIVE/TRANSMIT STATUS - The receive and transmit status regis- 
ters may also be both read and written. Writing the RESET bit 
causes the particular half of the interface (receive or trannies) 
to reset itself. If that portion of the interface is active when 
the reset is initiated, the operation in progress will be aborted, 
the error bit in the end address register will be set, and the 
receive or transmit level for the device will be written to 

the PID. Before writing the begin address, bits 1-3 of the 

buffer beginning location must be written into bits 0-2 of the 
corresponding status register. Reading one of the status registers 
allows a processor to determine the PID level associated with 
that direction of data transfer and to interrogate the QUIT flag. 
The PID will be written and the QUIT flag will be set if a QUIT 


occurred during the previous data transfer performed by the DMA. 
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This could indicate a parity error, non-existent address, etc. 
In this case, when the end pointer is read, the error bit will be 


set. 


The interpretation of the device dependent status bits varies 
from device to device but in general these bits provide for 
Girect: Gwo-way communication between: a processor and a device 


interface. 


One of the device dependence bits. will be the ACTIVE. Div. which, 
if set, indicates that a transfer to or from the device is in 
progress. More precisely, a DMA device is active from the time 
that its end pointer is written (which starts the device) until 
the time that it writes its level to the PID ING Leatvi ne Ut as 


done). 


DEVICE DEPENDENT - This register can be optionally used by the 
device interface for any appropriate function. The assignment 


of data bits is arbitrary. 
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To cause a transfer to be performed by a BBN DMA device, the 
program will typically perform the following steps: 
1. Write the STATUS REGISTER - This sets up the low-order 3 bits 
‘of the buffer start word address and selects any desired options 
(e.g., looped modem). This will normally be done only once for a 
sequence of DMA transfers. 
2. Write the BEGIN ADDRESS REGISTER - This sets up the 16 high 
order bits of the buffer start address. | 
3. Write the END ADDRESS REGISTER - This sets the end address of 
the buffer and initiates the DMA transfer. 
When the PID level, indicating device completion, is picked up by 
a processor, it will: 
4, Read the END ADDRESS REGISTER and check bit 15 (completion). 
If it is set, the transfer has completed (i.e., no error occurred 
and this is the last buffer of the transfer). Bits 1-12 are 
used to give the length of the buffer. 
5. If this bit is not set, bit @ (error) is checked. If bit @ 
is zero, then no error occurred but this buffer is not the last 
of the transfer. As above, bits 1-12 are used to determine the 
length of the buffer. 
by 16 pat Bis one, then an error has occurred. These are dif- 
ferentiated by examining the STATUS REGISTER. If bit 13 (active) 
is set, the device is still active and the PID value was spurious. 
If bit 8 (QUIT) is set, a QUIT occurred during the transfer. 


Device dependent status bits may further define the error. 


In addition to the registers mentioned above, each BBN block » 
transfer type of device has a number of manually settable switches. 
These switches, located on the device interface, are as follows 
(number of switches provided for each purpose shown in paren- 
theses): 


(i) Device Address Switch (10) - These switch settings define 
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the address of the device register block in I/O space (see 
Figure 5). The ten switches specify bits 4-13 of the address of 
the first register of the block. (Bits 14-19 of this address 
are all ones and bits 0-3 are all zeros.) 


(17) Receive/Transmit PID levels (7) - These seven switches 
define the number written to the PID upon completion of a data 
transfer. For duplex devices there are two sets of switches. 
For simplex devices only a single set is provided (e.g., CBT - 


see section 10.5). 


(iii) PID Address (2) - Selects which of the 4 PIDs will be 
written to by the device. The selected PID must be on the same 


Infibus as the device itself. 


Civ) Device Number (8) - In general, a set of 8 switches readable 
as the low-order byte of the first word in the device register 
block (see Figure 7). The CBT device, however, has only one such 


Switch. 
6.3 BBN Non-DMA I/0 Devices 


Typically, non-DMA devices will only have a small amount of 
internal hardware buffering, therefore, they need to be serviced by 
a processor no slower than every few byte times. The mechanism by | 
which such a device is serviced can take one or two forms in Pluribus 
systems. One approach is to let the device be passive and put the 
responsibility for servicing the device completely on the processors. 
For input, the processors would have to poll the devices faster than 
the input rate so that no data is lost. For output, the processors 
would have to deliver data to the devices at a rate sufficient to 
guarantee that no undesirable gaps within the data occur. Although 
such an approach permits a relatively simple hardware interface im- 
plementation, it may require an undesirable amount of processor 


overhead. 
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111111 | 10-BIT DEVICE ADDRESS SWITCH | 9000 
weR6 —~ yp 18 87 g 
—— 8-BIT DEVICE NUMBER 
a DEVICE TYPE a) veer eernr oe |... SWITCH ON. 
| DEVICE INTERFACE 
| 15 | 


0 
‘ HIGH ORDER 16 BITS OF | 
FIRST BUFFER LOCATION 
. RECEIVE 
STATUS . | 
, | TRANSMIT LAST BUFFER OF TRANSFER 
BEGIN ADDRESS io (TRANSMIT ONLY ) 19 
| LOW ORDER 13 BITS | 
5 WRITE OF BUFFER END ADDRESS* 
BITS 1-12 OF LAST 
6 READ WORD TRANSFERRED ADDRESS 
7 DEPENDENT sali 
| | NO ERROR & LAST BUFFER OF 
TRANSFER (RECEIVE ONLY ) 
15 987 3 2 Q 
LOW 
wre] (A ons 
| _ DEVICE TG, “Yj ADDRESS 
L. -DEPENDENT-- ! 4 
status |@ | 
READ UY PID LEVEL lo 
1 


_ * BIT © ON TRANSMIT END HAS A SPECIAL INTERPRETATION 
FOR THE CBT DEVICE (SEE SECTION 10.5) | 


** ONLY ONE SWITCH EXISTS FOR CBT DEVICE (SEE SECTION 10.5) 


Figure 7 DMA Registers 
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An alternate approach is to make the device active with respect 
to notifying the processors when it requires service. Ina Pluribus 
system this implies that the device will write its level to the PID 
when its internal buffers are ready. Checking whether the device 
needs service will, therefore, be done automatically as part of 
the main PID reading loop of the program. Such an approach, of 
course, requires more hardware in the device interface than does 


implementation of the first approach mentioned above. 


The only BBN non-DMA I/O device which currently exists is 
the synchronous line interface (SLI) which is described in detail 
in section 10.7. This device is passive and consequently requires 


polling by the processors. Both DMA and non-DMA I/0 devices which 
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are addressed through system address space will have 16 byte device 


register blocks associated with them. In contrast to the DMA 
device register blocks which have a common format for all DMA 
devices, the structure of the non-DMA device register blocks will 


be device dependent. 
6.4 Lockheed SUE I/0 Devices 


As indicated above, standard SUE I/O devices differ from 
those developed specifically for the Pluribus system in that (1) 
they interpret 16-bit addresses rather than 20-bit addresses and 
(2) if they are set up to actively notify the processor when they 
require service, they do so via a hardware priority interrupt 
mechanism rather than via the PID. Since it will often be desirable 
to incorporate such devices in a Pluribus system, some procedures 
for interfacing and programming them need to be developed. There 
are two distinct approaches that may be taken. First, sufficient 
modifications could be made so that the device will work on the 
system I/O busses. This approach has the advantage that the I/0 


device will be accessible to any processor in the system. It has 
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the disadvantage that hardware modifications probably need to be 
made to the device hardware. The other approach is simply to have 
the LEC device reside on one of the processor Infibusses, the place 
for which it was designed. This approach has the disadvantage of | 
essentially isolating the device from the processors in the system 
on other processor busses but has the advantage of requiring no 
hardware modifications. | | 


If the first approach is taken, that is, the device is put on 
an I/O bus, the hardware modifications required depend on whether 
the device will be active or passive. In either case it will be 
essential to modify the device interface to recognize ec0Q-bit ad- 
dresses. If it is to be a DMA-type device it would also be required 
TO generate 20-bit addresses. In the ARPA Network application, for 
example, a system control teletype has been interfaced in this manner 
and is handled by the processors as a passive device. Programming 
of LEC devices on the I/O busses will be similar to the BBN I/0 
devices discussed above. The details, of course, depend on how the 


device interfaces are modified. 


The only logical difficulty with putting LEC I/O devices on 
the I/O busses arises in the case of high speed DMA devices which 
require fast servicing for proper operation. To guarantee that 
such devices will be serviced within a specified time is likely to 
impose unacceptable constraints on the size of the strips into 


which tasks are partitioned. In such cases, e.g., handling a disk, 


it will probably be essential to take the second approach mentioned 


above and interface the device on one of the processor busses. 


Programming LEC I/0 devices on a Pluribus processor bus is 
essentially identical to programming them in a standard multipro- 
cessor SUE configuration. The only difference arises with DMA 


devices which are dealing with buffers in Pluribus common memory. 
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Since the devices only produce 16-bit addresses, some mapping 
mechanism similar to the processor address mapping is required 
here. This can be accomplished by simply dedicating one of the 
first three map registers associated with processor 0 to the I/O 
device for the duration of the time the device is being used. 
The I/O device addresses which have no key bits set appear to 
the bus couplers as requests from processor O and are mapped 
accordingly. I/0 interrupts, when they occur, are always routed 
to processor 0. Even though it is undestrable from an overall 
system reliability standpoint, this dependence on a specific 


processor is unavoidable. 
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More detail in programming specific LEC peripherals can be 
found in the LEC SUE Computer Handbook [2]. 
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7. SYSTEM RELIABILITY MECHANISMS 


The hardware architecture of Pluribus systems which provides 
a foundation for the development of reliable computer systems has 
already been presented. This section describes both some additional 
hardware mechanisms which have been included to improve system 
reliability and a general description of some software mechanisms 
which when operating on the Pluribus hardware can create a reliable 


computing environment in which to execute application programs. 


The interpretation of reliability is strongly related to the 
type of applications for which a computer system is intended. At 
one extreme are computations which do not have strict real-time 
constraints, for example, large numerical computations. For these 
applications, reliability may mean simply checkpointing, that is, 
dumping intermediate states of the computation on amass storage 
device so that the computation can be continued without much wasted 
effort should a system outage occur. At the other extreme are 
real-time control applications in which no outages are allowed and 
every request must be serviced within a short fixed time period. 
Such applications may require simultaneously running the system on 
several identical hardware configurations with decisions based on 
amajority vote. Although the Pluribus system can be applied to 
applications in both of these two classes, the applications for 
which it was specifically designed fall somewhere between these two. 
The Pluribus system will be most appropriate in situations where it 
is important to maximize MTBF and minimize MTTR but where occasional 


outages and minor delays in servicing requests can be tolerated. 


A reliable Pluribus system will generally be configured with 
at least one redundant copy of each nardware resource essential 
for running the overall system. It will be the goal of the relia- 


bility software to maintain a "working set" of resources and, if 
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possible, backup spares for each of them. In general, the relia- 
bility software will attempt to recover from failures of single 


hardware resources. 
7.1 Hardware Reliability Mechanisms 


The following mechanisms can be used both by the Pluribus 
reliability software described later in. this section and applications 


programs which choose to use them directly. 


7.1.1 Power Failure/Restart Interrupts: 


Cc 
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(i) Processor Infibusses - The Infibus provides power for — 
z) 

each Of The devices 2e-contains: .If the power supply ® 
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is about to fail on a processor Infibus, processor @ 
on that bus receives a level 4 interrupt with device 
code 2. Processor @ then has approximately 2.5 milli- 
seconds to signal the other processors and save any 
important data on a non-failing Infibus or in non- 
destructive memory. When a processor Infibus is with- 
Out Dower, The Control Register of any bus: coupler con-= 


necting this Infibus to another is still modifiable. 


When power is restored to the Infibus, a reset of the 
Infibus is executed by the BCU and each device on the 
bus will reinitialize itself. Each processor will 
enter an idle state with all registers zeroed. Proces- 
sor @ will then execute a level 4 interrupt with 


device code 4 (power restart). 


Cie Memory and I/O Infibusses - When power is about to 
fail on a common Infibus,; processor @ of each. infibus 
connected to the common Infibus executes a level l 


interrupt with device code 1. The processor must then 
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read the control register of each bus coupler on its 


Infibus to determine which one caused the interrupt. 


If the Control Register is 0*, then the attached 


Infibus is losing power. 


When the processor determines which common Infibus 
is losing power, it has 2.5 milliseconds to signal 
other processors, save important data stored on the 
failing Infibus, and mark that Infibus unusable by 
the program. While the Infibus is without power the 


bus coupler map registers are still modifiable. 


It will be necessary for the processors to periodi- 
cally check the dead Infibus to see if power has been 
restored. When this is the case, the Control Register 
will read (hexadecimal) 210@*. | 


Hardware Timeouts: 


A philosophy prevalent in the Pluribus hardware and 


software is that the system should perform sufficient 
introspection to recognize illegal and deadlocked states. 
If such states are detected, actions sufficient to move 
the system into some legal state should be initiated. 

The Infibus and device timeouts discussed below are two 


implementations of this general philosophy. 


*This may be modified by the contents of any overlapping memory 


location (see section 9.2). 
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hioleeel Infibus Timeout - The Bus Control Unit monitors the 
frequency of activity on the Infibus. If there are no 
accesses for one second, Che BCU will execute an Infibus 
RESET and each device on the Infibus will reinitialize 
itself. Processors on the bus will enter an idle state 
with their registers set to zero. Processor # will sub- 
sequently receive a level 4 interrupt with device code 4 


power restart. 


7.1.2.2 Device Timeout and Multiple Interfaces - In many applica- 
tions of the Pluribus it will be important to have redun- 
dant (multiple) interfaces to one or more of the I/O 
devices in order to improve system reliability. In the 


ARPA Network application, for example, multiple inter- 


Cc 
no) 
bud 

o 
— 

O 

W~—) 

® 
an) 


faces to modems and Host computers are planned. Since 
SUCH MUItiple “interfaces: will ‘share sa Single 1/0 device, 
it will be necessary to electrically disable the device- 
interface transmit path on all interfaces but the one 


currently 207 Use 26 any ¢iven tame. 


Rather than have the enabling/disabling of these 
Paths: Controlied by the processors, the a2nverraces 
themselves are provided. With sutiicient. Joeie so“ Unat 
the decision can be made locally. Each device interface 
which permit other interfaces like itself to be connected 
to a shared I/O device is equipped with a hardware timer. 
This timer is reset whenever a specific (device dependent ) 
word in the corresponding device register block 1s accessed. 
If the word is not accessed for a fixed time period, the 
timer runs to zero and the associated device - interface 
path is disabled. The path will be enabled whenever the 


specific word in the device register block is next 
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referenced. A processor can, therefore, switch from 

one interface to a spare by simply stopping references to 
one interface and starting references tO another. If for 
some reason an interface cannot be referenced, it will 
soon return to its stable, legal, disabled state. All 
DMA devices currently implement this facility and have 
one second timeouts. If a transfer is in progress when 
the interface timer reaches zero, the interface will 
abort the transfer, write its level to the PID, and set 
the error bit in the associated device register block. 
The specific words in the device register blocks which 
must be referenced in order to reset the timers are given 
in section 10, where the different I/O devices are dis- 
cussed. 


7.1.3 Remote Reference/Control of Devices on a Processor Bus: 


lolede Backwards Bus Coupling - Using the type of interbus 
communication described so far, it is not possible for 
@ processor on one bus to interact with a processor on 
another bus except by voluntary communication on the 
part of both processors through a mutually agreed upon 
portion of shared memory. A processor cannot directly 
halt, restart, or modify the registers or local memory 
of a processor on a different processor bus. To over- 
come this shortcoming, the Pluribus has an additional 
mechanism known as backwards bus coupling (BBC). Back- 
wards bus coupling permits requests to be transmitted 
to a processor bus via a bus coupler from an I/O bus. 
This is the reverse of the normal path from processor to 
I/O. This is illustrated in Figure 8a where a processor 
on bus A is trying to reference some address local to 
bus B via I/O Infibus C. 
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Backwards bus coupling from one processor bus to 
another is only possible over a shared I/O Infibus. In 
addition, only one bus coupler on an I/O bus can be enabled 

_, for backwards bus coupling at a time. Attempting to 
enable more than one BBC path on a single bus will produce 
unpredictable results. A lock will typically be as- 
sociated with this shared "BBC path" resource. 


For backwards bus coupling to proceed, the BBC enable 
bit in the Control Register of the bus coupler connecting 
the shared I/O bus to the target processor bus must be 
set to one. (See Figure 5) This can be accomplished by 
writing the hexadecimal constant DE7D to the control 
register if forward coupling is to be disabled or DE7F if 
forward coupling is to be enabled. The first 13 bits of 
these constants are a code word required to prevent indis- 
criminate modifications of the register by malfunctioning 
devices. After the BBC enable bit is set, the BBC Map 
Register (see Figure 5) may be set to point to the desired 
area of address space on the target processor bus (at- 
tempting to write the BBC map prior to enabling BBC will 
result in a QUIT). With BBC enabled and the map register 
set, reference tO locations on the target processor bus 

may proceed by making reference to corresponding bytes 
within the BBC window (see Figure 5). After BBC accessing 
is complete, the coupler control register should be re- 
turned to its previous state by resetting the BBC enable 
bit. 


Figure 8b illustrates the details of the BBC map- 
ntext of the situation shown in Figure 8a. 
To reference locations within the address space of the 


processor with key bits XY on bus B, processor A loads 
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bits 1 and 2 of the bus C BBC map register with XY and 
bits 3-15 of this map register with the high order 13 
address bits of an area on processor bus B. Subsequent 
references to bytes or words within the bus C BBC window 
are translated into 18-bit references on the target 


processor bus A at the address formed as follows: 


(low order 3-bits of 
address of byte 

referenced in BBC 
Window) 


Bits 3-15 of Map Register 


The BBC Map Register essentially serves as a base register 
which allows up to 8 bytes starting at the BBC map ad- 
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dress to be referenced. Of course, the BBC Map Register 
will need to be updated quite a bit if any significant 


number of BBC references are required. 


One further complication is the fact that simul- 
taneous forward and backward bus coupling requests conflict. 
The result of such conflicts will be short term deadlocks 
while the Infibusses at both ends of the bus coupler 

time out their respective requests prior to sending QUITs 
to the requesting devices. In Figure a, for example, if 
processor Po on bus A were attempting to access memory Mo 
on bus B at the same time that processor Po on bus B 
were attempting to access device D on I/O bus C, a 
deadlock would occur with respect to coupler BC. Busses 
B and C would, therefore, timeout the requests made to 
BC prior to sending QUITs to processor Py on bus B and 
bus coupler AC on bus C respectively. Since the time 
until a QUIT is returned is typically longer on a proces- 


sor bus than on an I/O bus, however, the bus coupler AC 
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will generally receive the QUIT first and terminate the 


BBC request passing on the QUIT to the requesting proces- 
. Sor, Ps on bus A. The forward bus coupling will then 


continue until completion. The point of this discussion 


is that the application program using the BBC mechanism 
Should be aware that QUITs may result, be prepared to 
test for them (see section 5.5.1), and repeat the BBC 
request if necessary. | 


| The following list summarizes the previous discussion with a 


typical sequence of steps to follow for BBC references: 


1) 


= 
3) 
4) 


Lock (or wait on lock) BBC path resource on I/O bus 
set BBC Enable bit in Control Register 
Write Map Register 


| Will involve sensing and reacting 
Set of BBC References: to QUITs and may involve changes 
| to map register 
Reset BBC Enable bit in Control Register 


Unlock BBC path resource on I/O bus. 


Remote Resetting of a Processor Bus - Writing a zero to 
bit @ of the control register of a bus coupler connecting 


a shared (I/O or memory) bus to another processor bus 


will cause that processor Infibus to execute a RESET. 


All devices on the processor bus will be reinitialized; 
each processor will enter an idle state with all registers 
zeroed. A subsequent 60 cycle clock interrupt (level 4, 


device code 1) will reactivate processor § on the bus. 


As with writing the BBC Enable bit, the first 13 bits of 
the word written to the control register must agree with 
the hexadecimal constant DE78.:° In addition, bits 1 and 2 
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of the control register should be written so as to create 
the proper state with respect to forward and backward 


coupling after the reset. 


ee eres eee Bus Amputation - Bus amputation provides a means of 
isolating selected active devices (I/O devices and proces- 
sors) from the remainder of a Pluribus system. In this 
way malfunctioning devices can be prevented from af- 
fecting the healthy system components. The unit of ampu- 
tation is the bus; that 18, a whole bus must be amputated 


if any device on that bus is to be disabled. 


Bit. 2. of the econtrol register of a. bus coupler must 
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be 1 if the coupler is enabled for forward bus coupling. 


By setting this bit @, therefore, all forward requests _ 
over the coupler can be blocked. By zeroing this bit in 
all bus couplers coming from a bus containing a malfunc- 
tioning device, that device can be removed from the 


operational Pluribus. system. 


A processor 1S: -not: able To write the «control regisvers 
of any couplers connected to its bus (see section 9.2), 
therefore, amputation must be accomplished by references 
to the Control resister arrivine over a different: path. 
In Figure 9, for example, coupler ac could not be shut 
oft by Processor Eas on bus A. Processor P on bus B, 
however, could cut path ac with an access to the ac 
control register (in the address space of bus C) via 
eoupler be AS With writing the REESE! and: BBC. Enable viLts, 
the first 13 bits of the word written to the control 
register must agree with DE78 hexadecimal, therefore, the 
word to write to the control register to amputate a bus 
is DE79 if BBC is to be disabled and DE7D if BBC is to 
be enabled. | 
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6. 
PROCESSOR PROCESSOR B 
| BUS | BUS 


bd 


MEMORY ¢ 
BUS 


cd 


Figure 9 Bus Amputation Example 
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To illustrate, suppose that in Figure 9 processor 
oe on bus B decided that processor Po on bus A was mal- 
functioning and its bus should be amputated from the 
system. It could do this by simply zeroing bit 1 in the 
Control register Lor couplers ac and ad. Similarity, if 
processor ae determined that device X on I/O bus D was 
writing spurious data bits to common memory, it could 
isolate the device by zeroing bit 1. in the control 
register for coupler cd. This would effectively remove 
bus D from the system as far as memory transfers are 
eoncerned although addresses on bus D could still be 


referenced via couplers ad and bd. 
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7.1.4 Externally Initiated Reloads: 


FOr Uhe ARPA Network application it was necessary to 
develop a means of reloading Pluribus systems remotely over. 
phone lines. In other communication applications, the ability 


LOGO: Chis: may also be 1mporvant. 


A special piece of hardware called the Reload (RLD) 
Device is available which resides on an I/O bus and monitors 
up to 8 modem interfaces (receivers). When the RLD observes 
a command in the. input stream, it interprets the next 20 bits 
as a system address and the following 16 bits as a data word 
to be stored at that address. The address and data are heavily 
checksummed. Sequences of such commands can be sent to cause 
the RLD-déevice to fill a portion .of common. memory or wrive via 
backwards bus: Coupling to the processor bus adqress: spaces. 
Details concerning line protocol and device operation are 


given in section 10.6. 
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7.1.5 Parity Generation/Checking: | 


Parity generation and checking schemes provide a simple and 
effective way to detect many of the errors which occur in computer 
systems. The Pluribus uses such a scheme to automatically recognize 
memory failures and failures along the data transfer paths (i.e. bus 
couplers). This mechanism is invisible to the programmer except for 
the fact that a QUIT may result from a data access if bad parity is 
detected. 


The type of parity calculated is called "address XOR Data" 
parity or AXD parity for short. AXD parity involves two parity 
bits for each 16-bit word, one associated with each 8-bit byte. 
Bach parity bit is calculated as the exclusive-or of the address 
parity and contents parity of the byte. The advantage of this 
parity function is that it detects: (1) one data bit in error, 
(2) all data bits zero, (3) all data bits one, and (4) one address 
bit in error. | | | 


There are essentially four distinct paths ina Pluribus system 
That implement parity checking. Each of these paths involves inter- 
bus transfers and is described below. Parity checking for data 


accesses on a single bus is not implemented. 


(i) Processor/Common Memory Path - When data is being written 
TO common memory the processor end of the bus coupler computes 
the AXD parity bits and sends them to be stored in the memory. 
On reading a memory location the stored parity bits are re- 
trieved and returned to the processor end of the bus coupler 
which recalculates the parity and matches it against the re- 
trieved bits. A QUIT will be generated if these two sets of 
parity bits do not match. | 
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(ii) I/0 Device/Common Memory Path - The procedure here is 
virtually the same as for the Processor/Common Memory Path 
above except that the I/O end of the bus coupler rather than — 
the processor end checks the parity bits. 


(iii) Processor/I/0 Device Path - When a processor writes 
(reads) data to (from) one of the devices on an I/O bus, a 
different sort of parity checking is performed. A special 
device on the target I/O bus, the Parity (PAR) card, con- 
Cinuously generates AXD parity from addresses and data placed 
on its bus during accesses to devices on that bus. This parity 
is fed back through the bus coupler involved in the reference 


and checked against parity computed at the processor end. A 
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QUIT will be generated if the two sets of parity bits do not 
match. This technique is referred to as feedback parity 


checking. 


(iv) Backwards Bus Coupling Path - Parity checking during BBC 
references is restricted to the forward part of the overall 
processor bus-to-processor bus path. The BBC registers are 
treated by the PAR card as if they were the registers of an 
I/O device on that bus, consequently the parity checking 


described in (iii) above applies. 


7.1.6 Transfers Between Private Memories On the Same Processor Bus 


Using the BBC mechanism it is possible for a processor on one 
processor bus to transfer data into the private memory of a proces- 
sor on another processor bus (see section 7.1.3.1). It is also 
desirable for a processor to be able to effect transfers into the 
private memory of another processor on the same processor bus. 

Such transfers will be required, for example, when a processor on 


a bus wants to reload and restart another processor on that bus. 
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A recommended technique for doing this is described below. It 
involves running a short program out of the registers in one of 
the processors. 


Suppose processor O wishes to transfer N words from its private 
memory starting at location SOURCE to consecutive words starting 
at location DESTINATION in processor 1's private memory. Processor 
O first stores the following program in processor 1's registers 
(starting at FF20): 


LDA A2, SOURCE (-Al1) 


KEY 1 

OTA A2, DESTINATION (Al) 

KEY 0 

BNLP FF20 /Address of Processor 1 register @ 
J MP (AT) 


Next, Processor O sets up the count N in one of his registers (Al 


in the example above) by executing the following: 
LDA Al, = N 


Finally, processor O executes the program on processor 1's registers 
via a: 


JSB AT, FF20 
This example has assumed, of course, that processor 1 was initially 


halted and that the original contents of processor 1's registers 


either did not matter or were initially saved and later restored. 


| An attempt by the reader to work out some more straight forward 
solution should demonstrate the necessity of the sort of implemen- 


tation described above. 
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7.2 Software Reliability Mechanisms 


There are no strict constraints on the programmer concerning 
how the Pluribus hardware features can be used. These hardware 
mechanisms have been developed, however, with a particular hardware/ 
software structure in mind. This structure will be described below. 
It should be pointed out that there does not currently exist any 
reliability software package that is available with a Pluribus 
system. The Pluribus reliability software which now exists is 
integrated into the ARPA Network IMP application of the Pluribus. 
Nevertheless, we believe that the basic ideas embodied in this 
software (and perhaps much of the code itself) can be applied in 


other Pluribus applications andare, therefore, worthwhile describing 
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here. 


One view of the relations between the three major software 
components ina reliable Pluribus system is shown in Figure 10. 
From. the figure 20. ‘can be seen, that. ‘there are: two major ‘modules oF 
the reliability software. The system reliability code is applica- 
tion independent and attempts to maintain a suitable set of re- 
sources in which to run the overall system. The application relia- 
bility code, on the other hand, is totally dependent on the particular 
application since it has responsibility for checking and fixing 
Che data structures internal to the application program. To. 
develop this module one must have a detailed knowledge of the 
states of that program. For this reason, the following discussion 


will focus on the structure of system reliability code module. 


Under normal circumstances, the application program will be 
continuously running, executing application tasks fetched from the 
PID. The system timer routine which runs off of the real-time 
clock (RTC) causes both the application reliability code and the 


system reliability code to be periodically executed. The system 
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reliability code is comprised of a sequence of stages that are 
performed when activated. These stages include such tasks as 
calculating the checksum on programs in local and common memory, 
checking whether any memory or I/O device has either appeared or 
disappeared, maintaining original and spare copies of code and 
variable segments, and maintaining the running status of all 
processors by reloading and restarting them if necessary. If all 
these tasks can be performed successfully, the system reliability 
software will return to the application program. This will normally 
be the case. In some situations, however, the system reliability 
code may be required to supervise the initialization of the applica- 
tion program itself. Reinitialization of the application code, 
would be required, for example, if a segment of memory containing 
variables were taken out of service and a new portion of memory 


were allocated for this purpose. 


An important concept associated with the system reliability 
module is that of processor consensus. Before a processor is 
allowed to run either the application program or the application 
reliability code, it is necessary to establish a common environment 
for all processors. This process of reaching an agreement about 
the environment is called "forming a consensus", and we dub the 
group of agreeing processors "the Consensus". The work done by 
the Consensus is in fact performed by individual processors, but 
the coordination and discipline imposed on the Consensus members 
make them behave like a single logical entity. An example of a 
task requiring consensus is the identification of usable common 
memory and the assignment of functions (code, variables, buffers, 
etc.) to particular segments. The members of the Consensus may 
not agree in their view of the environment, as for example when a 
broken bus coupler blinds one member to a segment of common memory. 


In this case the Consensus, including the processor with the broken 
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coupler, will agree to run the main system without that processor. 


In addition to periodic activation by the system timer routine, 
the system reliability code will also be activated following certain 
exceptional conditions indicated in Figure 10. Several of these 
conditions have already been discussed. An extremely important 
mechanism not yet mentioned, however, is the 60HzZ interrupt which 
is used to guarantee that each processor does, in fact, periodically 
run the system reliability code. Each processor upon executing the 
system reliability code sequence will reset a timer which the 60Hz 
interrupt service will count down. If the timer ever reaches zero, 
a processor has been lax for one reason or another and the relia- 


“Sility code will try to get the processor running correctly again. 
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As is the case for periodic activations, the system reliability 
code will eventually either go to sleep or supervise the initiali- 
zation of the application reliability routine or the application 


program itself. 


The discussion in this section has only provided a brief 
overview of the Pluribus software reliability mechanisms which are, 
in fact, currently in state of flux. More details and additional 
motivation for many of the design decisions relating to Pluribus 


reliability mechanisms may be found in [7]. 
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Figure 10 Reliability Software 
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8. INFIBUSSES 


The Infibus is the primary power and communication path- 
way between devices. Physically, the bus is a panel containing 
24 slots. Each device is inserted into one or more of these slots. 
Power and signal circuits connect all of the slots together. 
Power for the bus is provided by one of two possible power supplies: 
the smaller power supply is plugged into 8 of the slots of the bus, 
leaving 16 slots for devices; the larger power supply is external to 
the bus (leaving 24 slots for devices) and can provide power for 
one or more busses (depending on power requirements of the devices). 
The Bus Control Unit (BCU) module is necessary to control every 


Infibus. It occupies one slot, leaving either 15 or 23 slots for 
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other devices. A bus can be extended by the addition of another 


bus cabinet. The electronics for the extension will occupy one 
slot in each cabinet, leaving 29, 37, or 45 slots for devices. The 
number of slots occupied by the major components of the Pluribus 


system are as follows: 


Device Number of Slots 
Processor 2 
8k bytes Memory 3 
16k bytes Memory 3 
Bus Control Unit (BCU) 1 
Bus Coupler (BCP, BCM, or BCI - see section 9) 1 
PID Pseudo Interrupt Device 1 
RTC Real Time Clock dy 
HLC Local Host Interface 2 
CBT Checksum/Block Transfer 2 
ML Low Speed Modem Interface 3 
RLD Reload Card 1 
PAR Parity Module 1 
SLI Synchronous Line Interface 1 
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Electronically, the Infibus is the communications channel 
between devices. At any time, at most one device contained ina 
bus has access to that bus. This device can request data from 
another device contained in the bus (read) or request that another 
device receive the data that this device is providing (write). 

The device which has access to the bus is called the bus Master. 
The Slave device is the device the bus Master is transferring data 
to or from. The Master communicates with the other devices con- 


tained in the Infibus by providing the following information: 


26-bit Address 
One of the control functions: 
Read 
Write 
Read-Modify-Write 
Whether data is word or byte 
- Data (if function is Write) 


Parity 


Rach device contained in a bus continuously monitors the address 
being transmitted by the bus Master. A device becomes the Slave 
when it observes an address on the bus that it recognizes as its 
own. The device then performs the activity indicated by the ad- 
dress and control functions. When this activity is completed, a 
completion signal called DONE is returned. When the Master observes 
the DONE signal it accepts any data expected from the bus and 
relinquishes access to the bus. The Bus Control Unit has at that 
time already chosen the next device to be Master from among the 
devices which have requested access to the bus but have not yet 
received it. If no device recognizes the address that the Master 
provides the bus or if the Slave device malfunctions, then no 
action will be taken and no DONE signal will be returned. After 
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a fixed period of time (dependent on the particular bus), the 

Bus Control Unit will send a QUIT signal to the bus Master. The 
bus Master then relinquishes control of the bus and access is pro- 
vided the next requesting device. The time allowed between access 
and a QUIT signal is established by the BCU hardware and is nor- 
mally between 5 and 5@@ microseconds. Processor busses will 
normally have the longest QUIT timeouts with I/O busses and memory 


busses having the next longest and shortest timeouts respectively. 


Two different devices on a bus can recognize the same address. 
If both of these devices respond with action and a DONE, the system 
will likely malfunction. Devices must, therefore, use some criteria 


external to the bus to resolve which device becomes the Slave. 
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Normally the address recognition switches on each device in @ sys- 
tem will be set to recognize disjoint portions of the system ad- 


dress space. 


The: DUS, prPOVIdEeS. an inivialt zation Siena. callbed' RESET... vo 
each of the devices attached to it. This signal is transmitted to 
the devices whenever power is being restored, whenever the bus is 
reset from the console or from another processor, or whenever there 
nas. Deen: nO. transaction -on: this bus. Ion the last second, Hach 
device will terminate any activity when it receives the RESET signal 


and reinitialize the state of all registers and indicators. 
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9. Bus Couplers 


The functions of the bus couplers as components in an opera- 
tional Pluribus system have already been discussed in several 
earlier sections. In this section the internal structure of the 


bus couplers is considered in more detail. 


Kach bus coupler connects two busses and, therefore, has two 
ends. Each end of a bus coupler appears as a normal device on its 
containing Infibus. The 3 types of ends (BCP, BCM, and BCI) and 
two types of bus coupler (BCP-BCM and BCI-BCM) that may exist in 
a Pluribus system are illustrated in Figure 11. BCP-BCM couplers are 
used to connect processor busses to either memory or I/O busses. 
BCI-BCM couplers are used to connect I/O busses to memory busses. 
The operation of the BCP, BCM, and BCI devices are presented below. 


9.1 BCP: | 
Each BCP contains four 7-bit map registers for each of the 

four possible processors on the Infibus. The map registers are 
numbered 0-3 and are located in the address space of each processor 
at locations FC0%-FC#6. Each processor has its own set of map 

registers, selected by bits 16 and 17 of the 18-bit address of 
data on the Infibus. These two bits are specified by the last 
execution of the SKEY of JKEY instruction in the particular processor. 
The map registers can be modified by writing the new contents of 
the map to the corresponding address FC0®, FC®@2, FCO4, or FCLE. 
The high order 7 bits of the word written become the new contents 
of the map register. In general, bus coupler registers may be 
written but not read. Reading a map register gives a result of 
zero and does not change the register. Infibus RESET does not 
effect the contents of the map register. The contents of the map 


registers are unpredictable at power-up. 
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Figure 1] Types of Bus Couplers 
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During forward (normal) bus coupling the BCP is a Slave 
device on its bus and the BCP transforms each 18-bit processor 
address into a 26-bit System address to be sent to the BCM. As 
discussed in section 4., each processor's address space is divided 
up into 7 components: | 


Addresses | Description 

OOOO-3FFF . References to Local Memory 

4 $00-SFFF Transform address using map 0 
6000-7FFF Transform address using map 1 
SOOC-IFFF Transform address using map 2 
AGOO-BFFF Transform address using map 3 
CQ00-FBFF | Transform address to I/O space 
FCOQO-FFFF _ | References to Processor Registers 


and Local I/O space 


Addresses within {900-3FFF or FCQQ-FFFF are ignored by the BCP. 

For those 8k byte segments of processor address space utilizing 

a particular map, the BCP forms a system address by preserving the 
low order 13 bits of the processor generated address while replacing 
the high order 3 bits by the 7 bit contents of the corresponding 
map register. Addresses in the segment COQ0-FBFF are considered to 
be references to Pluribus device registers. The system address 

for such a reference consists of appending four bits of 1's to the 


most significant portion of the address. 


When the BCP transforms an address, this address, any 
data, and one of the control operations (read, write, read-modify- 
write, byte) are communicated from the BCP to the attached BCM 
through a cable. The control operations will be used by the BCM to 
bus 
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the bus access on the source bus except for the transformation of 


the address. Read operations using map 3, however, will be 
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transformed as previously described into read-modify-write accesses 
on the target bus (where the write data is zero) to allow imple- 


mentation of multiprocessor locks. 


When backwards bus coupling is enabled, the BCP acts as a 
Master on its bus and simply passes along the 18-bit references 
generated by the BCM at the other end of the cable. 


9.2 BCM: 

When a processor accesses a shared resource on @ memory or 
T/O bus, all of the BCPs on the source bus map the initial address 
and pass it along to the BCM end of the bus coupler. Similarly, 
when an I/O device accesses a shared resource on a memory bus, 
all the BCIs on the source bus transmit the initial address to 
their BCM end. Each BCM then determines if the address sent to it 
is one to which it can respond. If it is not, the BCM simply 
ignores the request. If it is, the BCM requests access to its 
Infibus. When it receives control, the BCM transfers the 2@-bit 
address, any data, and all control signals to its bus and returns 
any responses received to the originating end of the bus coupler 
pair. The addresses to which a BCM will respond are determined by 


the Cable Recognition Switch described below. 


There are two important reasons for making the bus coupler 
perform address discrimination. The first is to reduce hardware 
contention. If each BCM simply passed all addresses to the con- 
taining bus, every processor reference to common memory would be in 
contention for each memory bus rather than just the single bus on 
which the referenced memory was located. A similar contention 
problem would exist for processor references to I/0 busses and I/O 
references to common memory. The second reason for BCM address | 
discrimination is to eliminate multiple responses by the connected 


busses. Since a bus always responds either positively (by DONE) or 
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negatively (by QUIT), one DONE and multiple QUITs would result from 

every access to common memory if no address discrimination was 

done. The QUITS would, of course, confuse the device that previous- 
ly requested the access since it would already have taken actions 

based on the previous DONE. This same problem is the motivation 

for configuring Pluribus systems so that BCMs connected to different 
busses recognize disjoint areas of system address space. In general, 

the addresses recognized by all BCMs connected to the same bus will 

be identical. 


The BCM contains several physical switch registers which must 
be manually set and a single 16-bit control register which may be 
referenced under program control. The switch registers along with 


the number of bits (switches) in each register are indicated below: 


Switch Register Number of Bits 


MEMSW (Memory or I/O Bus) 


BCM CONTROL REGISTER ADDRESS 6 
BCM ADDRESS RECOGNITION: 
BASE 8 
RELEVANCE 


The algorithm used by the BCM for address discrimination is as 
follows: if the 20-bit address it receives is less than FCO, 
then the high order 6 bits of the address are compared against 

the high order 6 bits in the BCM ADDRESS RECOGNITION switches. 

The comparison is satisfied for a particular address bit if either 
the corresponding RELEVANCE switch is OFF or the RELEVANCE switch 
is on and the address bit matches the corresponding switch (bit) in 
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20-bit address is accepted and used to request 
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Typically, the BASE AND RELEVANCE switches will be set to recognize 
a contiguous portion of system address space. This is done by 
setting the high order 6 bits of BASE to some starting address and 
turning off some number of low order switches (within the high 
order 6) in RELEVANCE. Of course, more complicated memory access 
patterns can be implemented by other settings of the RELEVANCE 


Switches. 


If the 2@-bit address passed to the BCM is greater than or 
equal to FC@ZG and MEMSW is on, the address will not be recognized 
by the BCM. If the address is greater than or equal to FC@00 and 
MEMSW is off, bits ll and le of the address must satisfy the com- 
parison test described above with respect to the two low order 
bits of the BCM ADDRESS RECOGNITION switches if the 28-bit address 


is to be recognized and put on the (I/0) bus*. 


The BCM contains one internal register, the BCM Control 
Register. As already indicated in section 4.3 and 6.1, the block 
of addresses where these control registers can be found is at the 
beginning of one of the address space segments recognized by the 
bus to which the BCMs are connected. The precise location of a 
BCM Control Register is specified by the 6-bit BCM CONTROL REGISTER 
ADDRESS Switch. The number set in this switch is used as the dis- 
placement in words of the BCM Control Register from the starting 
address of the control register block. To state this more suc- 


cinctly, the address of each BCM control register is: 


Early models of the bus couplers required all 8 high order bits 
of the address to match the switch bits for address recognition 


to occur. 
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Address Bits From 


ileal OO High order 6 bits of BCM ADDRESS 
RECOGNITION BASE switches 

13 | | Negation of MEMSW 

(18 0 | 

1-6 | Contents of BCM CONTROL REGISTER 
ADDRESS switches. 


0 ) 


The 3 switches (BBC Enable, Forward Enable, and Reset) 
which can be set in the BCM control register by a processor have 
already been discussed in detail in section 7.1.3. It has also 
been pointed out that the data written to a BCM control register 
must agree with the high order 13 bits of the hexadecimal constant 
DE78 if the write is to take effect. | 


One additional complexity in the BCM arises since the Control 
Registers for BCMs on a memory bus and those on an I/O bus will 
differ in one respect; those on a memory bus will share addresses 
with the memory devices on that bus whereas those on an I/O bus 
will not share locations with any I/O device. Since devices 
referencing BCM control registers will expect a single DONE signal 
upon completion of an access, the BCM works as follows: if MEMSW 
is on, the BCM does not return a DONE since references to the 
control register also reference a memory location and the memory 
device returns the DONE signal. If MEMSW is off, on the other 
hand, the BCM generates and returns a DONE signal for references 
made to its control register since there is no other "overlapping 
devices" that will produce it. 
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The effect of two devices (BCM and memory) sharing the same 
address must also be kept in mind when the BCM control register is 
read or written. For BCMs attached to I/O busses, reading will 
return 2100 if the attached bus is up or 0 if the bus is in the 
process of going down due to a power failure-- see section 7.1.1 
(of course a QUIT will be returned if the attached bus is completely 
down). If the BCM shares the address of its control register with 
a memory module, however, this 2100 or O will be Inclusive-Or'ed 
with the contents of the associated memory word. For this reason 
and to permit proper operation of the Pluribus system parity 
mechanism, any read of a BCM control register will normally be. 
preceded by a write of zero to the control register. This will 


clear any "shadow" memory location but not affect the control 
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register contents since the password is incorrect. The response to 
stores at the address of the BCM control register will also depend on 
whether the address is on an I/O bus or a memory bus in addition to 
whether or not the high order 13 bits of the data written match the 
key DE78. If the Key matches, the write will take effect and a DONE 
will be returned. If the Key does not match, a DONE response will be 
returned if the BCM control register is on a memory bus and a QUIT 


response will be returned if the BCM control register is on an I/O bus. 


9.3 BCI: 


The BCI serves in place of a BCP when coupling an I/O Infibus 
to a memory Infibus. Its relation to The BCM is identical to that 
of the BCP with the following three exceptions: (1) no address 
mapping is performed (devices on I/O busses generate 20-bit addres- 
ses), (2) any address less than FC0Q@ (with any data and control 
Signals) is passed directly through the BCI to the BCM, and (3) any 
address greater than or equal to FC@@G is ignored. Devices on an 


I/O Infibus cannot directly communicate, therefore, with I/0 
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devices on another I/O Infibus; any such communication must be 
done via common memory. The BCI-BCM bus coupler cannot be used 


for backwards bus coupling. 
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10. DEVICES 


In section 6, the general information necessary for program- 
ming both BBN-developed and LEC devices was discussed. This 
section provides additional device-dependent information for each 
of the BBN-developed devices. Similar information for Lockheed 


devices can be found in the LEC Product literature. 
10.1 Pseudo Interrupt Device (PID): 


The PID is a priority memory device. The application of this 
device to the control of processes ina Pluribus system has been 
described at length in section 5. A Pluribus system may have up to 
four independent PIDs. As indicated in Figure 5, the PID and 
Real-Time Clock (RTC) share a 16-byte device register block in the 
address space of the containing I/O bus. The three registers in 
this block associated with the PID are the following: 


PID WRITE: 


When data is written into the PID WRITE register, bits 
1 through 7 of the data get a zero appended as bit 0 
and the resulting even 8-bit number is stored. Only 
one copy of any number is retained. When the PID WRITE 
register is read, the largest number stored in the PID 


is returned but not deleted. 
PID READ: 


When the PID READ register is read, the largest number 
Stored in the PID is returned and that number deleted 

from PID storage. If there is no number stored in the 
PID, a zero is returned. An attempt to write the PID 

READ register will result ina QUIT. 
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PID CLEAR: 


When data is written into the PID CLEAR register, bits 

1 through 7 of the data get a zero appended as bit O and 
the resulting even 8-bit number is compared with the 
memory of the PID. If that number is already in PID 
memory, it is deleted. When the PID CLEAR register is. 
read, the largest number in PID memory is returned but 
not deleted. 


The address of each PID is specified by a two bit switch on the 
device which selects bits 11 and 12 of the device register block 
starting address. Bits 0-10 of the device register block starting 
address are zeros and bit 13-19 are ones. The PID card has a set 
of lights which display the high order 7 bits of the largest 


number stored. 
10.2 Real-Time Clock (RTC): 


The Pluribus has two methods for timing events. Processor @ 
on each processor bus can receive an interrupt every 1/60 of a 
second on processor interrupt level 4. This rate is too infrequent 
for many applications, however, and does not fit into the proces- 
sor independent structure of the Pluribus. For these reasons 
another timing device, the RTC, has been developed. The RTC also 
provides a common time reference for all the processors. The RTC 
provides access to a clock which is incremented every 100 micro- 
seconds and which generates two distinct PID levels periodically, 
one every 1.6 milliseconds and another every 25.6 milliseconds. 


As indicated in Figure 5, the RTC and PID share a 16-byte 
device register block in the address space of the containing I/0 
Infibus. The five registers in this block associated with the RTC 
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are the following: 


CLOCK COUNTER: The 16-bit clock counter, The RTC increments 


this register every 100 microseconds. 


CLOCK PID LEVELS: The high-order byte is the number which will 
be written to the PID every 25.6 ms. The low-order 
byte is the number which will be written to the PID 


every 1.6 ms. 


CLOCK READABLE REGISTER 1 - A switch-settable register. 
CLOCK READABLE REGISTER 2 - A switch-settable register. 
CLOCK READABLE REGISTER 3 - A switch-settable register. 
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These five registers are all read-only. Attempting to write to 


them will have no effect. 


The RTC has a device timer which will stop the clock if it 
has not been accessed (i.e. none of its 5 registers have been read) 
within the past second. This allows the Infibus timeout mechanism 
to.detect and recover from the illegal state where the RTC is the 
only device putting requests on an I/O Infibus. The clock will 
also stop on master (bus) reset. Reading any RTC register will 


start the clock again in these two cases. 


The address of each RTC is specified by a two-bit switch on 
the device which selects bits 11 and l2 of the device register 
block starting address. There is also a two-bit switch which 
specifies the address of the PID to be written to in the same way. 
Normally, these two bits will be identical to the two bits which 
locate the RTC device register block. In any case, bits 0-10 of 
the device register block starting address are zeros and bits 13-19 
are ones. Two seven—bit switches are also available on the RTC 


to specify the 1.6 second and 25.6 second PID levels. 
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10.3 Low Speed Modem Interface (ML): 


The ML connects the Pluribus to a 33 modem at speeds up to 
250kb. The ML will transmit or receive messages of an arbitrary 
even number of data bytes to or from the 363. The Pluribus need 
only specify a portion of the message to the ML at any time. This 
portion of a message is called a buffer and is delimited in core 
by two addresses provided to the ML by the Pluribus DMA. On 
transmission, the ML will transmit end of message characters if an 
end of message flag is associated with a buffer. On reception, a 
buffer will be read until either the input area provided is full 
or the end of message characters are detected. Three test options 
are available on the ML under program control: (1) the ML can be 
erosspatched to itself so that it takes its transmitted data back 
in as receive data ignoring the 303 modem, (2) the ML can loop 
the 383 modem back thus testing both the ML and the 383 modem, and 
(3) the ML can be forced to send a zero checksum to test the error 
logic of the ML. 


To allow for multiple MLs connected to the same modem, the 
device timeout feature described in section 7.1.2.2 has been im- 
plemented for the ML. Data buffering is provided on the ML card 
to tolerate delays in servicing of approximately 32 characters on 
both input and output without loss of data or line utilization. 
Additional delays of indefinite length are tolerated on output by 


sending a line protocol idle sequence. 


All data on the communication line is organized as 8-bit 
bytes, and sent low-order bit first. There are four control 


bytes: 
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NAME CODE (Hexadecimal ) 
SYN 16 
Dis 10 
STX 02 
HTX 83 


The protocol on the line looks as follows: 


oo DS CT D:D i De. 2.78 De eG ee ss 
E E E 
YY iL.7 X LL X Ij x | Lae Ose Orme Y Y¥ : 
NNE X aT EE iT en | Pi a xk 2: B NN rat 
Oo 
iL 2 3 4 3 5 3 6 i @ 
a) 
Notes: 
deg At least two SYN characters separate adjacent messages. 


An idle line is filled with SYN characters. 


Os The beginning of a message is indicated by the sequence DLE STX. 
The following character is text. Note that if the RLD card 
is used, the first bit of text must be zero if the message is 


not to be interpreted as a special reload message. 


ce The text is made up of characters taken from the buffer to be 
sent. The right half word is sent first, then the left half- 
word. There are always an even number of text characters in 


a valid message, otherwise messages are of arbitrary length. 


te, When the escape character DLE appears in the text, the hardware 
inserts an additional DLE. 


oe If text is not available to the ML in time, the sequence DLE SYN 


is sent as an idle protocol until text becomes available. 


83 


Report No. 2930 Bolt Beranek and Newman Inc. 


6. The end of a packet is indicated by the sequence DLE ETX. 


7. Each packet is followed by a 24-bit CRC checksum, sent as 3 
8—-bit characters. The checksum is computed based on all of 
the characters in the message starting with DLE STX and 
ending with ETX. A bad checksum will cause the device 


receiver to report an error. 


Note that a DLE is always followed by a DLE, SYN, STX, or ETX. 

Any other character following a DLE will cause an error on receiving. 
An error may also be reported if the ML receiver is not serviced 
quickly enough, that is, within 64 character times of the time that 
it writes the PID. A receive reset command flushes the input buffer 
and forces the receive half of the interface to look for character 
Sync. Detected errors also reset the interface to search for charac- 
ter sync, and flag the data as end of packet and error, but do not 
flush the buffer. 


The ML is a DMA device and as such is programmed as described 

in section 6.2. That section also describes the switches and device 
independent bits in the ML (DMA) registers. The interpretations of 
device dependent bits in the registers are indicated below. In | 


each case, the description assumes that a one is read or written. 
DEVICE TYPE: The high order byte contains a l. 


RECEIVE STATUS (15) "Loop": | 
Write: Cause the 383 modem to send back the transmission 
of the ML to the receive portion of the ML. The 
receive portion of the ML should be initialized 


before the transmit portion so that no data is lost. 


Read: The ML is looped. 
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RECEIVE STATUS (14) "Crosspatch": 
Write: Cause the ML to take back its transmitted data 
into the receive portion ignoring the 383 modem. 


The receive portion of the ML should be initialized 


before the transmit portion so that no data is lost. 


Read: The ML is cross—-patched. 


RECEIVE STATUS (13) "Active": 
Read: The ML receive portion is either waiting for or 


transferring a buffer from the 383 modem to memory. 


TRANSMIT STATUS (15) "Device Timeout": 


Read: A one second timer has deactivated the ML. If the 
transmit status word has not been written for one 
second, all activity of the ML is aborted. 

All ML circuits which communicate with the 33 
modem are deactivated. The ML will become usable 


again when the transmit status word is written. 


TRANSMIT STATUS (14) "Zero Checksum": 
Write: Generate a zero checksum for this message. 


Read: A zero checksum will be generated for this message. 


TRANSMIT: STATUS: (13) “Hepive™: 
Read: A buffer is being transmitted. 
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10.4 Local Host Interface (HLC): 


The Local Host module provides an interface between the 
Pluribus and another computer (called a Host) according to the 
hardware specification for IMP to Host connections described in 
the BBN report, "Specifications for the Interconnection of a Host 
and an IMP" [8]. This is a general purpose asynchronous serial 
interface. The Local Host module can perform block transfers of 
data in either direction between the Host and the PLURIBUS. A 
data block can be either read or written as a number of separate 
buffers if required. Transfer of the last buffer will have an 
associated end of data block flag. Padding is provided by the © 
HLC receiver at the end of data blocks to account for word length 
mismatch between the Pluribus and the attached Host. Two padding 
options are available: (1) a1 followed by O's as described in 
Report No. 1822 or (2) all zeros. The choice is fixed by hardware 
jumpers on the interface. Another set of jumpers permits the 
Pluribus and Host ready lines to be permanently disabled (ignored). 
The Local Host module can be programmed to be looped. In this 
state, all data transmitted from the transmit half of the HLC is 
returned to the receive half of the HLC. This mode of operations 
is convenient for hardware and software debugging. To allow for 
multiple HLCs connected to the same Host, the device timeout feature 
described in section 7.1.2.2 has been implemented for the HLC. 


The HLC is a DMA device and as such is programmed as described 
in section 6.2. That section also describes the switches and | 
device independent bits in the HLC (DMA) registers. The inter- 
pretations of device dependent bits in the registers are indicated 


below. In each case, the description assumes that a one is read 
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DEVICE TYPE: The high order byte contains a 2. 


RECEIVE STATUS C14) “hoop: 

Write: Connect the receive portion of the HLC to the 
transmit portion so that data transmitted by the HLC will be re- 
turned to the receive portion of the HLC. To initiate the trans- 
mission both RECEIVE END AND TRANSMIT END must be written. RECEIVE 
END should be written before TRANSMIT END so that no data is lost. 
When the HLC is looped, the HOST READY. indicator is the same as the 
Pluribus READY indicator. 


Read: The HLC is performing in looped mode. 
RECEIVE. STATUS: (1:53) “Aet ive”: 

Read: The receive portion of the HLC is active receiving 
a data block. 
RECEIVE STATUS. {l2); “Host Ready”. 

Read: A one indicates that the Host has set its ready 
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Dna LCa vor. 
RECEIVE soPATUS: -Cid)s 

Read: There was an error in the last buffer received from 
the Host. No end of message terminated the data block. This bit 
will be set if the Host Ready signal went away and returned during 
the previous transfer. Note that the error bit in the RECEIVE END 


register will also be set. 


RECEIVE, STATUS. €10:): 
Read: Same as RECEIVE STATUS (11) above except if this bit 
is set, an end of message indication did terminate the data block. 


TRANSMIT STATUS: "C1l4) “Loop: 
Read: The HLC is performing in looped mode. 


TRANSMIT STATUS: (13) “Active™: 
Read: The transmit portion of the HLC is active transmitting 
a data block. 
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TRANSMIT STATUS (12) "Pluribus Ready" 


Write: Writing a one sets the Pluribus ready indicator. 
Writing a zero clears the Pluribus ready indicator. This indica- 
tor will also be cleared if neither the transmit or receive status 
words has been written in the last second in order to implement 
the previously described device timeout feature. | 


Read: The Pluribus ready indicator is set. 
10.5 Checksum/Block Transfer Device (CBT): 


The Checksum/Block Transfer Device performs one of three 
operations on a source data buffer: (1) it calculates a 16-bit 
checksum on the data (2) it transmits the data words from the 
source buffer to a destination buffer, or (3) it does both (1) and 
(2) simultaneously. Aside from providing a convenient way to move 
data around with a Pluribus system, this device provides a key 
service for the system reliability software (see section 7.2). To 
the DMA, the device appears as two separate sections - source 
(transmit) and destination (receive) which deal with the DMA data 
transfer independently but are linked closely together within the 
device. Only the source interrupt and status, however, are used. 
Checksum calculation is performed serially, low order bit first 
with provision for either reinitializing or continuing the compu- 
tation when a new data block is specified. Either an IBM CRC 
16-bit checksum or a CCITT 16-bit checksum may be calculated. The 
choice is switch-selectable. The generator polynomials for these 


checksums are as follows: IBM: xi6 + x5 + x° + 1] and CCITT: 


x16 + xte + x? + 1. Transfer rate is limited only by bus access 
time when no checksum is being computed and by the slower of bus 
a 


.ccess or checksum computation when a checksum is being computed. 


Checksum computation time is approximately 1.3 microsecond per 
16-bit word. 


88 


Report No. 2930 Bolt Beranek and Newman Inc. 


The CBT is a DMA device and as such is programmed as described 
in section 6.2. That section also describes the switches and 
device independent bits in the CBT (DMA) registers. The interpreta- 
tions of device dependent bits in the registers are indicated below. 


In each case the description assumes that a one is read or written. 


DEVICE TYPE: The high order byte contains a 3. Switch (bit) 0 
selects a CCITT checksum (off) or an IBM CRC16 checksum (on). 


TRANSMIT (SOURCE) END (15): 
Write: This is the last buffer of the block. 


TRANSMIT (SOURCE) END (0): 


Write: Clear the checksum accumulator register. Writing 
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a zero indicates that the previous checksum should be pre- 
served, e.g. when checking a multi-buffer block. 


Read: Brror 


TRANGMIT -CSOURCE.) ‘STATUS: €15) “Check: 
Write: Calculate checksum. Changing this bit while an 
operation is in progress will cause an interrupt and a device 
reset. 


Read: Checksum being calculated. 


TRANSMIT (SOURCE) STATUS (14) "Transfer": 
Write: Move data from source buffer to destination buffer. 
Writing this bit while an operation is in progress will 
cause an interrupt and a device reset. 
Read: Data being moved from source buffer to destination 
buffer. 


TRANSMIT (SOURCE) STATUS (13) “Active”: 


Read: CBT operation in progress. 
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TRANSMIT (SOURCE) STATUS (12) "EOB Destination": 
Read: Interrupt requested Since the destination buffer 
is too small for source buffer. CBT has suspended activity 
until new destination buffer addresses are Supplied for the 
remainder of data. No data is lost. The error bit is also 
set. 


TRANSMIT (SOURCE) STATUS (11) "NOP": | oe 
Read: Interrupt requested since CBT initiated action but 
registers indicate that there is nothing to be done. The 
error bit is also set. | 


TRANSMIT (SOURCE) STATUS (10) "Last": 
Read: TRANSMIT (SOURCE) END (15) was set when this buffer 


was written. 


TRANSMIT (SOURCE) STATUS (9) "Destination QUIT": | 
Read: The receive (destination) portion of the device 
received a QUIT during the previous operation. The error bit 
is also set. | 


DEVICE DEPENDENT DMA REGISTER - The 16-bit checksum is accumulated 
here. This register may be initialized prior to the start of a 
checksum computation. Writing this register during a check opera- 
tion will cause an erroneous checksum to be calculated. 


10.6 External Reload Device (RLD): 

The Reload card (RLD) monitors the input data.from up to eight 
modem interfaces. When the RLD observes a command, it decodes the 
command as a 20-bit system address, a 16-bit data word, and a 16-bit 
CRC16 checksum. This single card device is not processor control- 
lable, but is controllabie by external signals arriving over the 
normal communication lines. The purpose of the RLD is to change, 


control, or restart the Pluribus system from a remote site without 
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on-site supervision. The RLD resides on an I/O bus, thus the RLD 
can modify common memory busses and access processor busses by 


backwards bus coupling as well as access devices on its own bus. 


Whenever a message is received over a communication line, 
the RLD checks the first bit (the least significant bit of the 
first 16-bit word). If this bit is one, the RLD determines that a 
sequence of RLD commands is arriving over that communication line 
and ceases to monitor the other 7 modem interfaces until all com- 
mands in the incoming message have been processed. Except for the 
first bit of the first command in the message, each of the remaining 
bits in each command are doubled to increase the uniqueness of the 
reload packet and to guarantee that the DLE character will not 
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occur in the reload data stream. The format of an arriving RLD 
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message is indicated below: 
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It should be noted that: 


1) The low-order two bits of the first word of the first command 
must be Ol. The low order two bits of the first word of 
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Subsequent commands in the message must be 00. 


2) An eight-bit padding byte follows the high order address bits. 

- This byte can contain any non-zero "doubled" four-bit pattern. 
The pattern can be set by jumpers on the card. (Note that 
four ones are shown in the figure above.) 


3) An arbitrary number of commands may contained in an RLD 
message. 


4) The 16-bit checksum on the undoubled address and data bits is the 


IBM CRC-16 checksum with generator polynomial xl® + x15 4+ x? +1. 


For each command in the message, the RLD device stores the 
incoming data word at the specified system address. This process 
is repeated until either a bad checksum is detected, bad padding is 
detected, non-doubled data is detected, a delayed bus access occurs, 
or the RLD device times out after one second of inactivity. In 
each of these cases, the RLD device releases the communication 


line and is available to service one of the other modem interfaces. 


There are 3 lights on the RLD device which provide a visual 
indication of the device operation. One light is on from the time 
that the device is first activated until the bus containing the 
device is reset. The second is on for the duration of a single RLD 
command sequence. The third indicator is briefly lit by completion 


of each bus access. 
10.7 Synchronous Line Interface (SLI): 


The SLI provides a simple synchronous full-duplex interface 
to a wide variety of modems. In contrast to the other interfaces 
previously described, the SLI is a single passive device and does 
not use either the DMA facility or the PID. To guarantee that 
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neither data nor line bandwidth will be lost, the processors must 
poll each SLI in the system faster than the byte rate being used. 


Each physical SLI card provides interfaces for two independent 
lines. The allocation of the 8 words in the device register block 
is given below. The location of the register block is fixed by 


jumpers on the card. 


Register 1: Device Type - Modem l 
Register 2: Status - Modem 1 
Register 3: Control - Modem 1 
Register 4: Data - Modem 1 
Register 5: Device Type - Modem 2 
Register 6: Status -—- Modem 2 
Register 7: Control - Modem 2 
Register 8: Data - Modem 2 


The Device Type and Status words are read only registers whereas 
the Control and Data words are read-write registers. The inter- 
pretation of bits in each of these four registers are given below. 
In each case, the interpretation assumes that the particular bit is 


one unless otherwise stated. 


DEVICE TYPE (8-15): The high-order byte of the Device Type 


register contains a 4. 


DEVICE TYPE (6-7): These bits are set by switches on the SLI 
card and indicate information concerning the speed of the 


modem to which the SLI is connected. 


Bits. (657) Speed 
00 Under 2.5 Kilobits/sec 
Ol Under 5 Kilobits/sec 
10 Under 10 Kilobits/sec 
11 19.2 Kilobits/sec and over. 
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STATUS (15): The transmitter buffer is empty. The next character 
may be written to the Data register. (It is assumed that 
Clear to Send status (10) - has previously been found to be on.) 


STATUS (14): A SYNC character has been transmitted. The program 
was too slow in responding to STATUS (15) above and in the 
absence of new data a SYNC character was transmitted. This 
bit remains set until the next data character starts being 
transmitted. 


STATUS (10): Clear to Send. This is a signal received from the 
modem. In general, this indicator signifies that the modem 
is ready to transmit data. Refer to the modem literature for 
more detail. 


STATUS (8): Data Set Ready. This is a signal received from the 
modem. In general, it indicates to the SLI that its modem 
is not in a test mode and its power is on. Refer to the 
modem literature for more detail. 


STATUS (0-7): After a Receiver Reset (CONTROL bit 0) this half of 
the Status Register will monitor the input data stream, that 
is, bits will be detoured here as well as going to the Data 
register. This will continue until a zero has propagated to 
STATUS 0O at which point these 8 bits will no longer change. 

A subsequent receiver reset will cause this first even char- 
acter search mode to start again. It is expected that this 
feature will be used to handle the case of devices transmitting 
to the SLI which employ different syne characters. The first 
even character received can by mutual agreement be the sync 
character that will be recognized by the interface hardware 
(see DATA (8) and DATA (9) below). 
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CONTROL (10): Request to Send. This signal is passed directly 
to the modem. In general, it indicates to the modem that the 
SLI is ready to transmit data. The modem will normally 
respond by setting Clear to Send STATUS (10). Refer to the 
modem literature for more detail. 


CONTROL (9): Loop Test. Loop the SLI output back into the SLI 
input. This feature allows the SLI to test itself without a 
modem. 


CONTROL (7): Transmit and recieve in 7-bit plus parity mode. 
This bit will be set by the program when communicating with 
an ASCII terminal. When writing to the data register bits 
0-6 will be accepted and the SLI hardware will add the correct 
bit: 7 to create odd parity in the 8-bit character transmitted. 
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Data words received will be checked for odd parity (see DATA 
(9) below) but bit 7 of the data byte read will be zero. For 
communication with EBCDIC terminals, CONTROL (7) is cleared. 
In this case parity is neither generated nor checked. The 
8-bit character transmitted is the 8-bit byte written to the 
data register. 


CONTROL (0): Receiver Reset. Clear Received Parity Error - 
DATA (9), Receiver Overrun Error - DATA (8), Syne Received - 
DATA (14), and Data Ready - DATA (15). As described above, 
writing this bit also initiates sync character search mode and 
initializes STATUS (0-7) to all ones. 


In contrast to the DMA devices previously described, the input 
and output halves of the SLI share a single address for the two 
(read and write) DATA registers. The proper SLI internal register 
is referenced when the SLI is accessed as described below. 
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DATA (15) 


Read: Data Ready. Finding this bit set signals that 

a new character is in bits 0-7 of the DATA register. 

If DATA (15) is zero, then no change has occurred to 
bits 0-7 of the DATA register since the last time the 
DATA register was read. Reading the DATA register 

sets bit 15 if it was one. In normal operation, the 
DATA register is read more frequently than the byte | 
rate, bit 15 is tested,and bits 0-7 extracted or ignored 
as appropriate. — | 


DATA (14) 


Read: Syne Received. This bit will be one from the 
time that a syne character is detected until a non-syne 
character is detected. Although available, this infor- 
mation will generally not be used by most programs. 


DATA (9) 
Read: A parity error has been detected. This bit is 
cleared only by Receiver Reset. It is never set unless 
CONTROL (7) has been set to one. Parity checking is 
enabled when data mode is entered, that is, when the 
first non-syne character after two successive sync 
characters arrives. 


Write: Store Transmit Syne. Route DATA (0-7) to a 
special holding register (rather than transmit it). 

This character will become the transmitted syne character, 
i.e. it will be transmitted whenever the last bit of 

a character has been sent but no new data character has 
been written to DATA (0-7). This register remains 
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DATA (8) 
Read: The incoming data stream has not been serviced 
quickly enough and a character has been lost. Since 
the input is double buffered, two byte times must have 
elapsed since Data Ready was last set for this to occur. 
This bit is cleared only by Receiver Reset. 


Write: Store Receive Sync. Route DATA (0-7) toa 
second special holding register (rather than transmit 
it). This character will become the receive syne 
character, that is, this character will be compared to 
the received bit stream to achieve character 
Synchronization. Data mode will be entered after at 
least two adjacent syne characters have been received. 


c 
2 
Goat 
oO. 
~ 
a) 
n 
® 
OQ 


This register will remain unchanged until rewritten. 


DATA (0-7) 
Read: Input Data Byte. This is the data to be extracted 
when Data Ready is Set. 


Write: Output Data Byte. This is the location to which 
the next 8-bit byte should be written when Transmit 

Buffer Empty is found set. This register is not protected 
against premature writing and no indication is provided 

if it is written when Transmit Buffer Empty is zero. 

If this happens, the character previously written will 
have been lost without being transmitted. Each receive 
and transmit portion of the SLI device is actually 

double buffered in addition to the serial-to-parallel 
shift register in the card. This extra buffering implies 
that the programmer actually has longer than a single 
character time in which to service the device. In addition, 


the programmer should also be aware that this buffering 
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has other implications since the contents of the status 
and data registers are not synchronized. A status 
indicator can not be associated with the data byte 
currently available in the data register. 


The SLI device can be used with either switched or 
dedicated channels. The Data Set Ready, Data Terminal 
Ready, and Carrier Detect signals will be useful primarily 
in switched applications. They can all be strapped to 
"true" values for unswitched operation. If the SLI 

is used with a full-duplex charinel (i.e. modem and 
circuit) the Request to Send and Clear to Send Signals 
could also be strapped "true". They are included to 

allow the option of half-duplex operation. 
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GLOSSARY 


Update History: 


Originally written by M.F. Kraley, February 1975. 
Revised by M.F. Kraley, March 1977. | 
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60 Hz. interrupt - a classical interrupt occurring at the 
power line frequency on level 4, device number l. 


abort - a QUIT. 


access time - time from the initiation of the request (rise 
of STRB) to the pececreatron or acknowledgment of data 
(rise of DONE). 


active - said of a DMA device while it is transferring data: 
from the writing of the end Bornes to the setting of 
the PID level. 


active I/O device - an I/O device which indicates its need 
for service directly, usually either by classical or 
pseudo interrupt; cf. passive I/O device. 


address halt - a feature of the control panel which halts a 
processor when ae selected address is accessed on the 
bus. 


address recognition - the process in which a module checks 
the Infibus address lines for an address which is in 
the range of those which pertain to that module. 


address space - the set of locations accessible to 
(addressable by) a device; cf. memory space, I/0 
space, system address space, processor address_ space, 
etc. | | 


amputate - to disconnect a bus (usually a processor. bus) 
from the rest of the system by turning off the forward 
enable bit in all bus couplers coming from that bus. 


ALD - a LEC card which implements the AutoLoaD function. 


arbitration - the act of choosing the next prospective user 


of a resource. 


ARPA Network - a national network of heterogeneous computers 
linked to facilitate research; the original design 
environment for the Pluribus. 


asynchronous - not necessarily occurring at a certain time 
or at fixed time intervals. 


asynchronous line - a serial communications line where’ the 
receiver derives timing information from the initial 
transition of a character's start bit; characters are 
sent individually, at arbitrary times , bounded by start 
and stop bits. : | 
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attention - a classical interrupt on level 1, device number 
FF80, caused by pushing the "attn" button on the 
control panel. | 


autoload - a LEC module which contains some read only memory 
programmed to do loading from any of a number of I/0 
devices; when commanded by a bus signal, the autoload 
will initiate a classical interrupt, having first set 
the vector address to point to the ROM. 


auto restart - see power recovery. 


auxiliary I/O space - the portion of I/O space from FC000 
through FDFFF. 


auxiliary processor - a processor which is not number 0 and 
thus does not handle classical interrupts. 


AXD parity - a scheme wherein the parity bit(s) are derived 
from both the address and data; specifically, the 
parity bit of each byte is the exclusive-OR of the odd 
parity of the data and the odd parity of the byte 
address of that byte. 


backwarde bus coupling - the process by which a master on a 
common (usually I/O or M/I) bus can access a slave on 
another bus (usually a processor bus); used by 
processors to access other processors' address space. 


bandwidth - the rate at which information may be transferred 
or processed. 


BBC - Backwards Bus Coupling. 


BBC enable bit - bit 2 of the bus coupler control register; 
controls whether that coupler is selected for BBC. 


BBC map - a register in the BCM which specifies’ the 
high-order address bits of a BBC reference. 


BBC window - the four word region of system. address space 
through which BBC references are performed. 


BBN -— Bolt Beranek and Newman Inc.; the developer of 
Pluribus. | 


BCI - Bus Coupler I/O end; the card which forms the I/0 end 
of an I/O-to-memory bus coupler. 


BCM - Bus Coupler Memory end; the card which forms the I/0 


end of a processor-to-I/O coupler and the memory end of 
processor-to-memory and I/O-to-memory bus couplers. 
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BCP - Bus Coupler Processor end; the card which forms’ the 
processor end of processor-to- memory and 
processor-to-I/O bus couplers. 


BCU - Bus Control Unit; a LEC card which performs’ bus 
supervisory functions; it is chiefly responsible for 
arbitrating the use of the bus, but also assists in 
classical interrupts and other specialized functions. 


BDR - Bus Driver/Receiver: a custom IC used in both LEC and 
BBN boards to interface with the Infibus. 


begin pointer - in a DMA device, the address of the first 
word of the buffer. | 


bezel -—- the decorative front of a bus unit which also 
- contains an air filter. 


block transfer - the act of copying the contents of a series 
of contiguous memory locations to another place. 


buddy - the other processor(s) on the same bus. 


buffer - a series of contiguous memory locations which holds 
a block of data. 


bus - usually an abbreviation for Infibus. 


bus arbiter - a BCU. 
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bus controller -—- a BCU. 


bus coupler - a module which allows transactions on one bus 
to be transformed into transactions on another bus, 
depending upon address; composed of a BCM, either a BCP 
or BCI, and connecting cables; performs other special 
features such as parity generation and checking, 
mapping, power isolation, and amputation. 


bus extender - a LEC module which allows one logical bus’ to 
span more than one bus unit; the extended bus looks 
just like one long bus; it consists of two cards, BXD 
and BXR, and two connecting cables. 


bus timer - a 1 second timer that is held off by bus accesses; 
upon timing out, resets the bus; guards against hung states. 


bus unit - the basic mechanical module of the Pluribus; contains 
various combinations of Infibusses and power supplies, and 
has integral cooling. 


BXD - Bus Extender Driver; the card which forms part of the bus 
extender; plugs into the same bus as the BCU. 


5 


a ee er : 


Report No. 2930 Bolt Beranek and Newman Inc. 


BXR - Bus Extender Receiver; the card which forms part of 


the bus extender; plugs into the bus which does not 
have a BCU. | oe 


byte - 8 bits; two bytes to a word. 


cable - an assembly which electrically connects two or more 
modules and/or external equipment; each type has a four 
letter designation. 


card - a logic board which plugs into the Infibus; each type 
has a three letter designation. 


CBT - a BBN card which forms part of a Checksum-Block 
Transfer module. 


CCITT checksum - a 16 bit checksum computed with polynomial 
x**16 + x**12 + x**5 + x*¥*0. 


CCP - Communications and Control Processor, a Pluribus 
application which involves the collection, limited 
processing and routing of seismic data. 


central processor - the number 0 processor electrically 
connected to the bus arbiter which handles all 
classical interrupts. 


checksum —- a number of bits associated with a block of data 
computed via a fixed function from the data; the 
implicit redundancy can then be used to detect changes 
in the data. 


checksum-block transfer -— a BBN module which allows the 
computation of a cyclic checksum and/or the copying of 
a block of memory; consists of a DMA and a CBT. 


classical interrupt - the diversion of the control stream of 
a processor in response to an external event; the 
device number of the interrupting device, status and 
program counter at the time of interruption are saved 
and the processor jumps indirect through a fixed 
location; also refers to the bus transaction which 
causes the interrupt. 


classical parity - the parity scheme wherein the source 
generates on writes and the source checks on reads. 


clock - usually refers to RTC. 


CMB - the LEC printed circuit board which forms the actual 
Infibus; holds the edge connectors for the cards. 
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common bus - a bus which is not a processor bus: a memory, 
I/O, or M/I bus. 7 


common memory - that memory which can be accessed by all 
processors, that is, all memory on memory or M/I 
busses. 


configuration - the process by which a group of Pluribus 
modules are selected and an arrangement designed to 
create a machine for a particular application. 


consensus - the agreement between processors that to take a 
particular action would be in their common interest; 
also refers to the process by which agreement is 
reached. : 


console - usually refers to the control panel. 


contention - the situation where multiple users are 
attempting to simultaneously use a resource; this 
usually causes delay. | 


continuous read/write - a feature of the control panel which 
when set, repeatedly performs the access requested by 
depressing the "read" or "write" buttons; located on 
the rear of the control panel. 


control panel - a LEC module which allows manual reading and 
writing of addresses, typically memory locations, 
processor registers, and starting and stopping of 
processors, and other special functions; consists of 
two cards, PCB and PBI, connected by a DIP connector 
cable, and a front panel, SWB, which connects to the 
other cards via three ribbon cables. 
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control register - a location associated with a module whose 
bits correspond to program settable functions; ina 
processor, register 15; in a serial or parallel 
interface, the address of the device + 6; in a bus 
coupler, set by the jumpers on the BCM. 


cooling module - the external shell of a bus unit which 
provides mechanical support for the Infibus chassis, 
fan pack, and bezel, and airflow isolation and 
deflection. 

CPA - a LEC card which forms part of the processor. 


CPB - an early LEC card which used to form part of the 
processor; superseded by CPC. | 


CPC - a LEC card which forms part of the processor. 
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CPU - central processor; more generally, but incorrectly, a 
processor. 


CRC-16 checksum - a 16 bit checksum computed with a ae 
| xX*¥*¥16 + x*¥*15 + x*¥¥2 + x**0. 


cycle time - the time from the beginning of a request until 
the device has completed all activity related to that 
request and is ready to start or accept BOT Pers 
usually longer than access time. 


cyclic checksum - a checksum computed by dividing the data 
by a specific polynomial and taking the remainder. 


D-cable - a cable which connects a card plugged into an 
Infibus with the fantail. | 


DBAL - Dual Bus Access Logic, a custom IC that contains much 
of. the logic necessary to be a bus master. 


DDT - a program which allows the user to inspect and change 
memory locations and processor registers, start and 
stop processors, copy memory, field traps and other 
useful things; in many ways, can be thought of as a 
simple executive, providing an environment for user 
programs. 


deadlock - a state in which two (or more) processes 
(hardware or software) are each waiting for a resource 
held by the other; each now waits indefinitely for the 
other's resource to become available. 


device - usually a module that performs I/O functions; 
sometimes refers just to DMA devices. 


device dependent - a register or bit whose interpretation or 
function is determined by the particular module with 
which it is associated. 


device independent - a register or bit with a common 
interpretation or function over a range of different 
module types. 


device register block - the eight word segment of address 
space which is associated with a DMA device. 


device type - a number indicating the type of the associated 
module; usually program readable in the low order byte 
of the first register of the device. 


device number - a number associated with each device which 
causes classical interrupts; when servicing an 
interrupt, this number can be read from the first word 
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of the interrupt vector, indicating which device caused 
the interrupt. 


DMA - Direct Memory Access; a BBN card which performs the 
bus interaction and pointer management for I/O devices. 


DMA device - an I/O device which uses a DMA; it transacts 
directly with memory with data in buffers. 


DONE - an MInfibus’- signal that indicates successful 
completion of a bus access cycle; also serves as the 
strobe for data in a read access. 


doubled cable - a cable which connects the two parts of a 
doubled interface with the fantail. 


doubled interface - an interface which, for reliability 
considerations, consists of two modules on different 
busses, connected such that either one can serve the 
external equipment. 


elastic buffer - a buffer which allows input and output to 
proceed asynchronously, at different rates. 


end pointer - in a DMA device, the address of the last word 
of a buffer. 


executive core - usually refers to locations O0-5E; the area 
in which interrupt, QUIT, and ILLOP information is 
stored. 


> 
dw 
0 
”n 

n 

2 

O 


EXY - Eight X and Y; a LEC card which forms part of the 
memory; contains the core stack itself. 


F-cable - a cable which connects external equipment to the 
fantail. 


fan pack - a chassis containing six fans that provide the 
cooling for each bus unit. 


fantail - a panel which contains connectors for cables from 
external equipment which interface to internal cables; 
used to facilitate the reconnection of external 


equipment. 


feedback parity - the parity scheme wherein the destination 
generates and the source checks parity on all 
transfers. 


flop - flip-flop. 


force reload — a scheme by which memory locations may be 
loaded, processors started, etc., via special,, heavily 
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passworded messages on modem lines; used for remote 
start-up of machines. 


forward enable bit - bit 1 of the bus coupler’ control 
register; when cleared, prevents all forward accesses 
through that coupler, thus amputating the bus 
associated with the BCP or BCI of that coupler. | 


F-stick - to map an address in I/O space via _ the implicit 
fixed mapping. 


full duplex - a communications path wherein transmission can 
take place in both directions simultaneously. | 


ground modem —- not a satellite modem. 

half duplex - a communications path wherein transmission can 
take place in either direction, but not both 
simultaneously. | 

halt(ed) - a state of the processor wherein instructions are 
not being executed, interrupts cannot be honored, and 
the registers are externally accessible. 

hex -—- abbreviation for hexadecimal, base 16. 

high speed modem - a BBN module which interfaces to a Bell 
306 modem at speeds up to 1.5 Mbaud; consists of MHX, 
MHR, and DMA. 


HLC —- Local Compatible Host; a BBN card that forms part of a 
Host interface. 


HIT - the name of the general Pluribus system test program. 


Host - a computer which provides and uses the actual network 
services; connected into the network via an IMP. 


Host interface - a BBN module which interfaces to a local 
Host; comprised of a DMA and HIC. 


hot code - frequently executed code which is located in 
local memory. 


IBM - four card interconnect module. 
IBM checksum —- CRC-16 checksum. 
ICM —- three card interconnect module. 


IDM -— two card interconnect module. 
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I-cable - a cable which connects two internal cards. 


idle - a state of the processor wherein no instructions are 
being executed, registers are not externally 
accessible, but interrupts may be honored. 


illegal operation - trap caused by attempted execution of an 
instruction not in the repertoire of the processor. 


ILLOP - illegal operation. 


ILLOP vector —- the four word block holding information 
pertinent to the current ILLOP; starts at 20 for 
processor 0, 30 for processor 1; contents are: illegal 
instruction, status, program counter, address’ of 
service routine. 


IMP —- Interface Message Processor; the node computer of the 
ARPA Network, which performs the basic packet-switching 
functions. 


Infibus - the bus which physically and electrically connects 
the cards of a Pluribus system. 


interface - a module which allows access and information 
flow to and from external equipment. 


interrupt - usually a classical interrupt. 
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interrupt vector - the four word block holding information 
pertinent to a given interrupt level; for levels 1-4, 
starts at locations 0,8,10,18 respectively; contents 
are device number, status, program counter, address of 
service routine. 


I/O bus - a bus which contains primarily I/O devices. 


I/O space - the part of system address space from FC0O00 to 
FFBFF; the area which may be accessed via fixed mapping 
from processor address space; also refers to the 
corresponding section of processor address’ space 
(COOO-FBFF ) 


isochronous line - a serial communications scheme wherein 
bit timing is derived from a separate clock line, but 
characters may be sent at arbitrary intervals and_ are 
bounded by start and stop bits. | 

jiffy - a 60 Hz. interrupt or 1/60th of a second. 


JIG -— the name of the bus coupler stand-alone test program. 
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K - 1024 (decimal). 


key bits - address bits 16 and 17, asserted on processor 

- veferences according to the contents of a two bit 

register set by the SKEY instruction; used to 
differentiate among the various processors on a bus. 


LEC - Lockheed Electronics Company. 
level 5 interrupt - an ILLOP. 
level 6 interrupt - a QUIT. 


local Host - a Host interconnected via a bit serial, 
handshook interface, usually over distances of less 
than 30 feet. 


local memory - memory on a processor bus, as opposed to 
common memory. 


lock - a data structure (usually a single word) used _ to 
| interlock processes; also refers to the act of reading 
a lock with a read-clear cycle. 


Lockheed Electronics Company - the manufacturer of several 
Pluribus parts. 


low speed modem interface - a BBN module which interfaces to 
a Bell 303 modem at speeds up to 250 Kbaud; consists of 
a MLX, MLR, and a DMA. | 


map value - the 7 bit number that determines which 4K page 
of system address space is referred to by accesses in 
the associated map segment. 


map segment - one of the four 4K regions of processor 
address space through which accesses are made to common 
memory. 


mapping - the act of transforming an address in one address 
space to that in another. 


master - the participant in a bus transaction which 
initiates the access; e.g. the processor, when it is 
accessing memory. 


memory - a LEC module, either 4K or 8K by either 16 or 18 
bits of random access core memory, consisting of three 
cards: TAG, SID, and EXY; also refers to a more generic 
collection of the above. 


memory bus - a bus which contains primarily common memory. 
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memory space - the part of system address space from 0 to 
FBFFF. | 


message - the unit of data communicated between Hosts; 
messages are broken up by IMPs into one or more packets 
for transmission in the subnetwork. 


M/I bus - a common bus which contains both Memory and I/O. 


MHR - a BBN card which forms the Receive half of a 
High-speed ground Modem interface. 


MHX -—- a BBN card which forms the transmit half of a 
High-speed ground Modem interface. 


MLR - a BBN card which forms the Receive half of a Low-speed 
ground Modem interface. 


MLX - a BBN card which forms the transmit half of a 
Low-speed ground Modem interface. 


modem - a piece of external equipment which converts digital 
Signals from the computer to analog signals for 
communication and vice versa; also refers to modem 
interface. 


‘modem interface -—- the module which interfaces to a high 
speed synchronous modem, either ground or satellite, 
low or high speed. 
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module - a unit consisting of one or more cards’ which 
performs a unified function. 


MSR —- the BBN card which forms the Receive part of the 
Satellite Modem interface. 


MST - the BBN card which performs the Timing functions of 
the Satellite Modem interface. 


MSX -— the BBN card which forms the transmit part of the 
Satellite Modem interface. 


multiprocessor - a system which contains several tightly 
coupled processors with some common resources. 


Multiwire - a technology for making cards, midway between 
printed circuit and wire wrap in the dimensions of cost 
and difficulty; consists of a printed circuit card 
which carries power and ground, covered by a sticky 
insulating layer, in which insulated wires are laid to 
form the signal paths. 
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P-cable - a cable which carries primarily power. 


packet - the unit of data communicated between IMPs on modem 
lines; several packets may form a message; usually on 
the order of one or two thousand bits. 


packet-switching - a communications scheme in which packets 
of data from many sources are forwarded to many 
destinations along the same line, multiplexing the use 
of the line at a high rate. 


page - a 4K region of common memory, or more _ generally, 
system address space. 


PAR - I/O PARity; a BBN card which generates parity for 
references to I/O devices. 


parallel interface - a LEC module that can interface up to 
20 parallel bits of information; can be polled or use 
classical interrupts; used primarily as the paper tape 
reader interface; card type PPB. 


parity - the exclusive OR of a collection of data and /or 
address bits; also refers to schemes which detect 
changes by generating and later checking the parity of 
a collection of bits. 


parity memory - memory which is 18 bits wide, allowing a 
parity bit to be stored for each byte. 


passive I/O device - a device which must be polled, does not 
interrupt; cf. active I/O device. 


password - a specific combination of data bits which must be 
written in order for an action to take place; used for 
reliability considerations. 


PBI - Panel Bus Interface; a LEC card which forms part of a 
control panel. 


PCB -— Panel Control Board; a LEC card which forms part of a 
control panel. 


PCD - PreCeDence passer; a BBN card which serves only to 
pass the precedence pulse by an empty slot; used for 
debugging. 


PDU - Power Distribution Unit; usually refers to ae BBN 

| module which accepts site power and distributes it, 

with appropriate switches, circuit breakers and 

indicators; also refers to a LEC module which provides 

two key switches, one for power, the other for 
processor selection. 
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PID — Pseudo Interrupt Device, a BBN module which serves as 
a hardware pending task queue. 


PID level - the number that a device or a processor writes 
to the PID to signify that the associated task should 
be run. 


Pluribus _ a line of modular, reliable, 
multiprocessor/minicomputer systems produced by BBN. 


poll - the act of periodically checking a device to see if 
some event has occurred, as opposed to the device doing 
its own notification when a change in status occurs. 


power fail - a classical interrupt which occurs 2.5 ms. 
before bus Operations are ceased preparatory to 
complete power loss; occurs on level 4, device number 


power restart - a classical interrupt which occurs’ on 
restoration of local bus power on level 4, device 
number 4. | 


power sense - an Infibus signal that indicates the condition 
of bus power, gives advance notice of a power failure; 
also refers to circuitry in the bus coupler that checks 
the status of power at each end of the coupler, 
allowing one end to disregard signals coming from a 
card with inadequate power. 
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power supply - a LEC module which supplies Infibus' logic 
power and a 60 Hz. Signal; comes in two styles: 
internal (plug-in, 5951) which takes up 8 of the 24 
slots of an Infibus, and external (stand-alone, 5952) 
which requires its own bus unit. 


PPB - Peripheral Parallel Buffer; parallel interface. 

precedence pulse - an Infibus signal which is daisy-chained 
between cards; used to resolve priority for the 
selection of the next bus master. 


primary I/O space - the portion of I/O space from FEO00O to 
FFBFF. 


printed circuit - a technology for fabricating cards which 
involves etching away copper-clad epoxy boards to form 
the signal paths. 


private memory —- local memory. 


processor - a LEC module which executes instructions; 
consists of two cards, CPA and either CPB or CPC; three 
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microcode versions exist: standard, business, and 
scientific. 


processor address space - the address space seen by an 
individual processor; 32K words long. 


processor bus - a bus which contains processors and 
susuarty? local memory. 


processor bus address space - the aggregate of the four 
potential processor address spaces on a processor bus; 
128 K words long. 


pseudo interrupt - the act of writing a number to the PID to 
indicate that a task associated with that number should 
be performed. 


PSB - peripnerat Serial Buffer; serial interface. 


QUIT - an Infibus signal that indicates abnormal completion; 
e.g. non-existent device, malfunctioning device, 
parity error, etc.; also refers to the trap that is 
taken when a processor-initiated access results ina 
QUIT. | | 


QUIT timer - the timer on the bus’) arbiter which regulates 
how long the bus will wait for a DONE before deciding 
that the intended slave will not respond and thus 
should issue a QUIT, terminating the access; these 
timers have different values on different bus types. 


QUIT vector - the four word block of memory which records. 
information pertinent to the most recent 
_ processor-initiated QUIT; contents are: address 
referenced, status, program counter, and address of 
service routine; located at 28 for processor 0, at 38 

for processor l. | | ee 


rack - the unit which houses bus units; up to five bus 
units, a PDU, and a fantail may be mounted in one rack. 


read - an access in which data is transferred trom slave to 
| DSC ee es 


read-clear - a read-modi fy-write access in which ae written 
data is zero. 


read-modify-write —- an access in which data is read from 
memory and then (potentially) different data is written 
back to the same address, all within one memory cycle. — 


real time clock - RTC. 
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reload card - RLD. 


remote power fail - a classical interrupt which indicates 
that a common bus's power is failing and about 2.5 ms. 
of usable power remains; occurs on level 1, device 
number l. 


remote reset - the low-order bit of a bus coupler's' control 
register; when cleared, causes a reset to occur on the 
bus which the BCP is plugged into. 


reset timer - see bus timer. 


resource - a part of a system needed by more than one of the 
parallel users and therefore a possible source of 
contention. 


ribbon cable - a multiconductor cable made of several 
parallel wires bonded together in a flat shape. 


run - a state of the processor in which instructions are 
being executed, interrupts may be honored, and 
registers are not externally accessible. 


ribbon - the part of an algorithm associated with a single 
PID level; may be one or more strips. 


RLD - ReLoaD; a BBN module which allows forced reloads. 


round robin - a feature of the bus arbitration scheme which 
enforces fairness of access to the bus; when a device 
has been granted a bus access that terminates normally, 
it is not allowed to request another access until all 
those devices on that bus which are currently 
requesting access have been granted same. 


RTC - Real Time Clock; a BBN card which causes PID levels at 
intervals of 25.6 ms. and 1.6 ms., has a program 
readable 16 bit counter that increments each 100 us., 
and three 16 bit readable switch registers. 


SACK timer - a timer on the bus arbiter which guards against 
the case wherein a potential bus master requests an 
access, subsequently stops’ the precedence pulse, 
indicating that it will be the next master, but fails 
to assert SACK, acknowledging this fact. 


satellite modem interface - a BBN module which interfaces to 
a satellite ground station transmitter; has features to 


enable use of the satellite channel in broadcast mode 


such as provision for switching the carrier and 
accurate timing of transmission and receipt of packets; 
consists of four boards: MSR, MST, MSX, and DMA. 
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scientific processor - the processor module with extended | 
instruction set, including multiply, divide, double 
precision, etc. | 


select cycle - that part of an access which is concerned 
with selecting the next master. 


self interrupt - a trap. 


serial interface - a LEC module that interfaces asynchronous 
(start/stop) I/O devices; strappable for various 
speeds, character sizes, EIA vs. current loop, modem 
options, etc.; is half duplex and may either be polled 
or use classical interrupts; card type PSB. | 


service cycle - that part of an access wherein the master 
actually transacts with the slave. 


SIMP - Satellite Interface Message Processor; an IMP which 
uses broadcast satellite channels as some of its 
communications links. 


simplex - a communications path wherein communication can 
take place in only one direction. 


slave - the participant in a bus transaction which responds 
to the master's request; e.g. the memory, when the 
processor is accessing it. 


SID -— Sense and Inhibit Drivers - a LEC card which forms 
part of the memory. 


SLI - Synchronous Line Interface - a BBN card which 
interfaces two synchronous lines; a passive device 
which must be polled. 


SMS - Synchronous Modem Simulator; a BBN card which 
interfaces two synchronous data sources, giving the 
appearance that the Pluribus is a modem; a passive 
device which must be polled. 


SRN - System Release Notice. 


status register - a location associated with a module whose 
bits report various combinations of the module; ina 
processor, register 8; in ae serial or parallel 
interface, the first register; in a DMA device, the 
fourth and seventh registers. 


step - the act of causing a processor to execute a single 
instruction and then halt. 
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start pointer - begin pointer. 


STRB - an INFIBUS signal which indicates that a master is 
transacting with a slave; used to strobe address and, 
on a write, data. 


strip - a set of instructions which are executed as a unit 
between references to the PID. 


SUE - System User Engineered; the name of the line of LEC 
parts which make up part of a Pluribus. 


SWB - SWitch Board; the front panel of the control panel. 


subnetwork - the collection of node computers (IMPs) and 
communication lines of a network which perform the 
actual routing and transmission of the data. 


synchronous - occurring at fixed time intervals. 


synchronous line —- a communications line where the timing 
information is derived from the transitions between 
data bits; data is usually sent in blocks. 


synchronizer - a device which resolves two asynchronous time 
references. 


system address space - the address space of common busses; 
seen directly by I/O devices and indirectly by 
processors after mapping; 512K words long. 
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system release notice - a document associated with each 
Pluribus system describing the location and 
configuration of each component. 


TAG - Timing And Gating; a LEC card which forms part of the 
memory. 


three phase wye - the type of AC power that the Pluribus PDU 
requires; there are five wires: one is for protective 
(green) ground; one is common for the other three, each 
of which has a 117 volt AC potential with the common, 
but the phase of the legs is staggered by 120 degrees. 


throughput - the rate at which information can be processed. 


TI - usually refers to Texas Instruments Silent 700 
terminal. 


timer - a device, hardware or software, which watches over 
the activity of a part of the system; the timer is 
periodically reset by the occurrence of an event which 
signifies correct operation and which’ should occur 
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periodically; should a specified time interval elapse 
without a reset, the timer will “time out" initiating 
some remedial action. 


TIP - Terminal Interface message Processor; an IMP with 
built-in simple Host capabilities which allows users at 
terminals access to the network, obviating an external 
Host computer. 


TOD -— Time Of Day; a modification to a pair of PPB boards to 
interface a Systron-Donner clock. 


trap - either a QUIT or an ILLOP. 

very distant Host - a Host connected to the IMP via a 
communication link, with associated error detection and 
retransmission protocols. 

VDH - Very Distant Host. 

VISTAR - refers to an Infoton VISTAR terminal. 

watchdog timer —- cf. timer. 

window - one of the four 4K regions of processor address 
space which can be mapped onto a page of system address 
space. 

wire-wrap - a technology for making boards wherein 
connections are made by wrapping a wire around a pin 


located adjacent to the component. 


word - the basic element of data; has 16 bits and two parity 
bits; there are two bytes in a word. 


woven cable - a multiconductor cable constructed by weaving 
together several twisted pairs with a nylon thread. 


write —- a bus transaction where the data flow is from master 
to slave. 
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